{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Investigate the Ice-Ocean Transition Across ICESat-2 Datasets\n",
    "\n",
    "## Motivating Questions\n",
    "- What does the ice-ocean transition zone look like across data products?\n",
    "- How does the presence of sea ice versus open water impact the profile?\n",
    "\n",
    "## Region of Interest\n",
    "- Greenland Marine Terminating Glacier\n",
    "\n",
    "## Code Resources\n",
    "- NSIDC DAAC ICESat-2 Customize and Access Jupyter Notebook by Amy Steiker from ICESat-2 Hackweek\n",
    "\n",
    "## Data Resources\n",
    "- ICESat-2 ATL06 and ATL07/ATL10 data from the NSIDC DAAC (https://doi.org/10.5067/ATLAS/ATL06.001)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import getpass\n",
    "import socket\n",
    "import json\n",
    "import zipfile\n",
    "import io\n",
    "import math\n",
    "import os\n",
    "import shutil\n",
    "import pprint\n",
    "import time\n",
    "import geopandas as gpd\n",
    "import matplotlib.pyplot as plt\n",
    "import fiona\n",
    "import h5py\n",
    "import re\n",
    "# To read KML files with geopandas, we will need to enable KML support in fiona (disabled by default)\n",
    "fiona.drvsupport.supported_drivers['LIBKML'] = 'rw'\n",
    "from shapely.geometry import Polygon, mapping\n",
    "from shapely.geometry.polygon import orient\n",
    "from statistics import mean\n",
    "from requests.auth import HTTPBasicAuth"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Downloading Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Earthdata Login password: ········\n"
     ]
    }
   ],
   "source": [
    "# Earthdata Login credentials\n",
    "\n",
    "# Enter your Earthdata Login user name\n",
    "uid = 'jessica.scheick'\n",
    "# Enter your email address associated with your Earthdata Login account\n",
    "email = 'jessica.scheick@maine.edu'\n",
    "pswd = getpass.getpass('Earthdata Login password: ')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "3"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "76EF5233-C93A-9E15-4AFB-5C9F0017258E\n"
     ]
    }
   ],
   "source": [
    "# Request token from Common Metadata Repository using Earthdata credentials\n",
    "token_api_url = 'https://cmr.earthdata.nasa.gov/legacy-services/rest/tokens'\n",
    "hostname = socket.gethostname()\n",
    "ip = socket.gethostbyname(hostname)\n",
    "\n",
    "data = {\n",
    "    'token': {\n",
    "        'username': uid,\n",
    "        'password': pswd,\n",
    "        'client_id': 'NSIDC_client_id',\n",
    "        'user_ip_address': ip\n",
    "    }\n",
    "}\n",
    "headers={'Accept': 'application/json'}\n",
    "response = requests.post(token_api_url, json=data, headers=headers)\n",
    "token = json.loads(response.content)['token']['id']\n",
    "print(token)\n",
    "#4351E4A7-9907-AEA1-82B3-FE2A36C88659"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Input data set ID (e.g. ATL06) of interest here, also known as \"short name\".\n",
    "\n",
    "short_name = 'ATL06'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Determine the number and size of granules available within a time range and location.\n",
    "\n",
    "#### Let's explore information about our data set. We'll start by determining the most recent version number of our data set. We will also find out how many data granules (files) exist over an area and time of interest. [The Common Metadata Repository](https://cmr.earthdata.nasa.gov/search/site/docs/search/api.html \"CMR API documentation\") is queried to explore this information. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'feed': {'entry': [{'archive_center': 'NASA NSIDC DAAC',\n",
      "                     'associations': {'services': ['S1568899363-NSIDC_ECS',\n",
      "                                                   'S1613689509-NSIDC_ECS',\n",
      "                                                   'S1613669681-NSIDC_ECS']},\n",
      "                     'boxes': ['-90 -180 90 180'],\n",
      "                     'browse_flag': False,\n",
      "                     'coordinate_system': 'CARTESIAN',\n",
      "                     'data_center': 'NSIDC_ECS',\n",
      "                     'dataset_id': 'ATLAS/ICESat-2 L3A Land Ice Height V001',\n",
      "                     'has_formats': True,\n",
      "                     'has_spatial_subsetting': True,\n",
      "                     'has_temporal_subsetting': True,\n",
      "                     'has_transforms': False,\n",
      "                     'has_variables': True,\n",
      "                     'id': 'C1511847675-NSIDC_ECS',\n",
      "                     'links': [{'href': 'https://n5eil01u.ecs.nsidc.org/ATLAS/ATL06.001/',\n",
      "                                'hreflang': 'en-US',\n",
      "                                'length': '0.0KB',\n",
      "                                'rel': 'http://esipfed.org/ns/fedsearch/1.1/data#'},\n",
      "                               {'href': 'http://nsidc.org/daac/subscriptions.html',\n",
      "                                'hreflang': 'en-US',\n",
      "                                'length': '0.0KB',\n",
      "                                'rel': 'http://esipfed.org/ns/fedsearch/1.1/data#'},\n",
      "                               {'href': 'https://search.earthdata.nasa.gov/search/granules?p=C1511847675-NSIDC_ECS&m=-87.87967837686685!9.890967019347585!1!1!0!0%2C2&tl=1542476530!4!!&q=atl06&ok=atl06',\n",
      "                                'hreflang': 'en-US',\n",
      "                                'length': '0.0KB',\n",
      "                                'rel': 'http://esipfed.org/ns/fedsearch/1.1/data#'},\n",
      "                               {'href': 'https://openaltimetry.org/',\n",
      "                                'hreflang': 'en-US',\n",
      "                                'length': '0.0KB',\n",
      "                                'rel': 'http://esipfed.org/ns/fedsearch/1.1/data#'},\n",
      "                               {'href': 'https://doi.org/10.5067/ATLAS/ATL06.001',\n",
      "                                'hreflang': 'en-US',\n",
      "                                'rel': 'http://esipfed.org/ns/fedsearch/1.1/metadata#'},\n",
      "                               {'href': 'https://doi.org/10.5067/ATLAS/ATL06.001',\n",
      "                                'hreflang': 'en-US',\n",
      "                                'rel': 'http://esipfed.org/ns/fedsearch/1.1/documentation#'}],\n",
      "                     'online_access_flag': True,\n",
      "                     'orbit_parameters': {'inclination_angle': '92.0',\n",
      "                                          'number_of_orbits': '0.071428571',\n",
      "                                          'period': '94.29',\n",
      "                                          'start_circular_latitude': '0.0',\n",
      "                                          'swath_width': '36.0'},\n",
      "                     'organizations': ['NASA NSIDC DAAC',\n",
      "                                       'NASA/GSFC/EOS/ESDIS'],\n",
      "                     'original_format': 'ISO19115',\n",
      "                     'processing_level_id': 'Level 3',\n",
      "                     'short_name': 'ATL06',\n",
      "                     'summary': 'This data set (ATL06) provides geolocated, '\n",
      "                                'land-ice surface heights (above the WGS 84 '\n",
      "                                'ellipsoid, ITRF2014 reference frame), plus '\n",
      "                                'ancillary parameters that can be used to '\n",
      "                                'interpret and assess the quality of the '\n",
      "                                'height estimates. The data were acquired by '\n",
      "                                'the Advanced Topographic Laser Altimeter '\n",
      "                                'System (ATLAS) instrument on board the Ice, '\n",
      "                                'Cloud and land Elevation Satellite-2 '\n",
      "                                '(ICESat-2) observatory.',\n",
      "                     'time_start': '2018-10-14T00:00:00.000Z',\n",
      "                     'title': 'ATLAS/ICESat-2 L3A Land Ice Height V001',\n",
      "                     'version_id': '001'}],\n",
      "          'id': 'https://cmr.earthdata.nasa.gov:443/search/collections.json?short_name=ATL06',\n",
      "          'title': 'ECHO dataset metadata',\n",
      "          'updated': '2019-06-20T21:51:48.635Z'}}\n"
     ]
    }
   ],
   "source": [
    "# Get json response from CMR collection metadata and print results. This provides high-level metadata on a data set or \"collection\", provide in json format.\n",
    "\n",
    "params = {\n",
    "    'short_name': short_name\n",
    "}\n",
    "\n",
    "cmr_collections_url = 'https://cmr.earthdata.nasa.gov/search/collections.json'\n",
    "response = requests.get(cmr_collections_url, params=params)\n",
    "results = json.loads(response.content)\n",
    "pprint.pprint(results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "001\n"
     ]
    }
   ],
   "source": [
    "# Find all instances of 'version_id' in metadata and print most recent version number\n",
    "\n",
    "versions = [i['version_id'] for i in results['feed']['entry']]\n",
    "latest_version = max(versions)\n",
    "print(latest_version)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2018-12-12T00:00:00Z,2019-02-28T23:59:59Z\n"
     ]
    }
   ],
   "source": [
    "# Input temporal range \n",
    "\n",
    "# Input start date in yyyy-MM-dd format\n",
    "start_date = '2018-12-12'\n",
    "# Input start time in HH:mm:ss format\n",
    "start_time = '00:00:00'\n",
    "# Input end date in yyyy-MM-dd format\n",
    "end_date = '2019-02-28'\n",
    "# Input end time in HH:mm:ss format\n",
    "end_time = '23:59:59'\n",
    "\n",
    "temporal = start_date + 'T' + start_time + 'Z' + ',' + end_date + 'T' + end_time + 'Z'\n",
    "print(temporal)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Area of Interest input"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "62.15,80.8,62.4,82.3\n"
     ]
    }
   ],
   "source": [
    "# Bounding Box spatial parameter in 'W,S,E,N' format\n",
    "\n",
    "# Input bounding box\n",
    "# Input lower left longitude in decimal degrees\n",
    "LL_lon = '62.15'\n",
    "# Input lower left latitude in decimal degrees\n",
    "LL_lat = '80.8'\n",
    "# Input upper right longitude in decimal degrees\n",
    "UR_lon = '62.4'\n",
    "# Input upper right latitude in decimal degrees\n",
    "UR_lat = '82.3'\n",
    "\n",
    "bounding_box = LL_lon + ',' + LL_lat + ',' + UR_lon + ',' + UR_lat\n",
    "# aoi value used for CMR params below\n",
    "aoi = '1'\n",
    "print(bounding_box)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Polygon coordinate pair spatial parameter\n",
    "\n",
    "# #create list of x (longitude) values in decimal degrees\n",
    "# x = []\n",
    "# #create list of y (latitude) values in decimal degrees\n",
    "# y = []\n",
    "# xylist = list(zip(x, y))\n",
    "# # Polygon points need to be provided in counter-clockwise order. The last point should match the first point to close the polygon. \n",
    "# # Input polygon coordinates as comma separated values in longitude latitude order, i.e. lon1, lat1, lon2, lat2, lon3, lat3, and so on.\n",
    "# polygon = ','.join(map(str, list(sum(xylist, ()))))\n",
    "# print(polygon)\n",
    "# # aoi value used for CMR params below\n",
    "# aoi = '2'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Use geopandas to read in polygon file\n",
    "# # Note: a shapefile or geojson, or almost any other vector-based spatial data format could be substituted here.\n",
    "\n",
    "# kml_filepath = str('/home/jovyan/data-access/pine_island_glims/glims_polygons.kml')\n",
    "\n",
    "# #Return a GeoDataFrame object\n",
    "# gdf = gpd.read_file(kml_filepath)\n",
    "# gdf.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [

    "Simple visualization of the polygon:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# %matplotlib inline\n",
    "\n",
    "# # Load \"Natural Earth” countries dataset, bundled with GeoPandas\n",
    "# world = gpd.read_file(gpd.datasets.get_path('naturalearth_lowres'))\n",
    "\n",
    "# # Overlay glacier outline\n",
    "# f, ax = plt.subplots(1, figsize=(12, 6))\n",
    "# world.plot(ax=ax, facecolor='lightgray', edgecolor='gray')\n",
    "# gdf.plot(ax=ax, cmap='Set2')\n",
    "# ax.set_ylim([-90, -30])\n",
    "# ax.set_xlim([-140,-40]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [

    "The following cell will simplify and reorder the GeoDataFrame object using the shapely package and convert the object back to a dictionary to be applied to the CMR polygon parameter. Simplification is needed in order to pass a reasonable request length to CMR. You may need to modify the simplification tolerance depending on the number of points of your polygon."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "POLYGON Z ((-86.622742 -74.908126 0, -86.561712 -74.870913 0, -86.868859 -74.73052199999999 0, -86.96290500000001 -74.60503799999999 0, -89.02594000000001 -74.316754 0, -89.630517 -74.19214700000001 0, -89.830808 -74.06591899999999 0, -90.746478 -73.95625800000001 0, -91.66821400000001 -74.023169 0, -92.049815 -73.92938700000001 0, -93.42079099999999 -73.929327 0, -93.997163 -73.882768 0, -94.27770099999999 -73.71418300000001 0, -95.133017 -73.96635499999999 0, -96.51350100000001 -74.127404 0, -99.889802 -74.085347 0, -100.114438 -74.01942200000001 0, -100.355131 -74.080906 0, -100.462734 -74.240864 0, -100.827076 -74.373988 0, -101.795349 -74.369597 0, -102.424826 -74.497263 0, -101.188725 -74.7179 0, -101.564382 -75.02970999999999 0, -103.37484 -75.273725 0, -103.914847 -75.426057 0, -104.012128 -75.5223 0, -103.029452 -75.748774 0, -102.350567 -75.749245 0, -101.837882 -75.943066 0, -101.899461 -76.01408600000001 0, -101.280944 -76.192769 0, -101.325735 -76.246168 0, -101.190803 -76.27106000000001 0, -101.250474 -76.342292 0, -101.175067 -76.345822 0, -101.402436 -76.52034999999999 0, -101.326063 -76.523929 0, -101.449791 -76.666392 0, -101.310795 -76.691373 0, -101.357407 -76.74481900000001 0, -101.217404 -76.769752 0, -101.295133 -76.85887 0, -101.058051 -76.96212300000001 0, -100.447336 -77.11768600000001 0, -98.43369800000001 -77.320866 0, -97.28308 -77.355688 0, -97.491148 -77.42317799999999 0, -96.514174 -77.485919 0, -96.552494 -77.55823599999999 0, -96.38465600000001 -77.562336 0, -96.44151599999999 -77.670857 0, -97.139363 -77.836566 0, -97.193451 -77.926901 0, -97.64270999999999 -78.080044 0, -96.29786900000001 -78.388943 0, -96.327803 -78.44329 0, -95.72146600000001 -78.511065 0, -95.74896200000001 -78.565482 0, -94.940425 -78.61707199999999 0, -94.98861100000001 -78.726066 0, -94.911669 -78.763976 0, -95.609268 -78.843079 0, -95.637038 -78.897535 0, -95.37191 -78.9391 0, -95.69340800000001 -79.006456 0, -95.269903 -79.124145 0, -95.323729 -79.233172 0, -95.430206 -79.249633 0, -95.15550500000001 -79.291032 0, -95.191045 -79.363748 0, -94.81352 -79.406486 0, -94.847075 -79.479253 0, -94.74744800000001 -79.48078 0, -94.772403 -79.53536699999999 0, -93.90411 -79.63884400000001 0, -93.84365099999999 -79.749409 0, -93.96732299999999 -79.802836 0, -93.788723 -79.87821 0, -93.81639300000001 -79.951128 0, -93.230546 -80.085534 0, -91.707475 -79.87748000000001 0, -91.801545 -79.822143 0, -91.48889699999999 -79.805457 0, -91.465152 -79.641131 0, -90.447349 -79.5894 0, -90.545492 -79.534464 0, -90.04231900000001 -79.37062 0, -90.140775 -79.33408300000001 0, -90.041814 -79.24285 0, -88.982186 -79.076903 0, -90.230262 -78.914333 0, -90.32191 -78.80480799999999 0, -90.689626 -78.67651600000001 0, -91.150024 -78.638589 0, -92.035347 -78.414844 0, -92.106013 -78.30491000000001 0, -91.651645 -78.271472 0, -91.365784 -78.127206 0, -91.188783 -78.128018 0, -91.09016699999999 -78.019109 0, -90.737076 -77.98384900000001 0, -90.90919100000001 -77.946905 0, -90.732603 -77.91100900000001 0, -90.72708799999999 -77.819973 0, -91.070502 -77.80062599999999 0, -91.14118000000001 -77.63646900000001 0, -91.90279 -77.613923 0, -91.984627 -77.595116 0, -91.97296299999999 -77.52236499999999 0, -92.466819 -77.463587 0, -92.199521 -77.374914 0, -92.352136 -77.30076099999999 0, -92.335283 -77.209895 0, -91.434206 -77.23465299999999 0, -91.42601500000001 -77.16193 0, -91.015545 -77.145686 0, -91.00835499999999 -77.054784 0, -91.08639700000001 -77.018096 0, -91.647835 -76.97871000000001 0, -91.640906 -76.924199 0, -91.873848 -76.86802400000001 0, -91.779021 -76.759619 0, -90.823937 -76.71007299999999 0, -90.345113 -76.52952999999999 0, -86.988029 -75.856983 0, -86.94556300000001 -75.71114300000001 0, -86.87223400000001 -75.710165 0, -87.034102 -75.63967 0, -86.96500399999999 -75.620616 0, -87.075115 -75.440545 0, -87.00315399999999 -75.439609 0, -87.021872 -75.349129 0, -86.835058 -75.21958600000001 0, -86.85065400000001 -75.14724699999999 0, -86.71772900000001 -75.10905200000001 0, -86.737771 -75.01866200000001 0, -86.602149 -74.99848299999999 0, -86.622742 -74.908126 0))\n"
     ]
    }
   ],
   "source": [
    "# #Integer position based indexing of GeoDataFrame object to get it into a shapeply geometry object.\n",
    "# poly = gdf.iloc[0].geometry\n",
    "\n",
    "# # Simplify polygon. The larger the tolerance value, the more simplified the polygon.\n",
    "# poly = poly.simplify(0.05, preserve_topology=False)\n",
    "\n",
    "# # Orient counter-clockwise\n",
    "# poly = orient(poly, sign=1.0)\n",
    "\n",
    "# print(poly)\n",
    "\n",
    "# #Format dictionary to polygon coordinate pairs for CMR polygon filtering\n",
    "# polygon = ','.join([str(c) for xy in zip(*poly.exterior.coords.xy) for c in xy])\n",
    "\n",
    "# # aoi value used for CMR params below\n",
    "# aoi = '3'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell provides an alternative option to post a file to OGR service for spatial file input conversion to CMR polygon format:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Alternative option for spatial file input: Post file to OGR service\n",
    "# # Spatial file input, including Esri Shapefile or KML/KMZ\n",
    "    \n",
    "# # POST shapefile or KML polygon to OGR for geojson conversion\n",
    "# url = 'http://ogre.adc4gis.com/convert'\n",
    "# shapefile = kml_filepath\n",
    "# files = {'upload': open(shapefile, 'rb')}\n",
    "# r = requests.post(url, files=files)\n",
    "# results = json.loads(r.content)\n",
    "# # Results is a dictionary representing a feature collection. List coordinates from the Polygon feature:\n",
    "# polygon_list = list(results['features'][0]['geometry']['coordinates'][0])     \n",
    "# # Remove z value from polygon list\n",
    "# for i in range(len(polygon_list)):\n",
    "#     del polygon_list[i][2] \n",
    "# # Create shapely Polygon object for simplification and counter-clockwise ordering for CMR filtering\n",
    "# poly = Polygon(tuple(polygon_list))\n",
    "\n",
    "# #Same simplify and orient steps as above:\n",
    "# #simplify polygon\n",
    "# poly = poly.simplify(0.05, preserve_topology=False)\n",
    "\n",
    "# # Orient counter-clockwise\n",
    "# poly = orient(poly, sign=1.0)\n",
    "\n",
    "# #Format dictionary to polygon coordinate pairs for CMR polygon filtering\n",
    "# # Polygon points need to be provided in counter-clockwise order as comma separated values in longitude latitude order, i.e. lon1, lat1, lon2, lat2, lon3, lat3, and so on. \n",
    "# # The last point should match the first point to close the polygon. \n",
    "# polygon = ','.join([str(c) for xy in zip(*poly.exterior.coords.xy) for c in xy])\n",
    "\n",
    "# # aoi value used for subsetting logic below\n",
    "# aoi = '3'\n",
    "# print(polygon)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CMR search parameters:  {'short_name': 'ATL06', 'version': '001', 'temporal': '2018-12-12T00:00:00Z,2019-02-28T23:59:59Z', 'page_size': 100, 'page_num': 1, 'bounding_box': '62.15,80.8,62.4,82.3'}\n"
     ]
    }
   ],
   "source": [
    "#Create CMR parameters used for granule search. Modify params depending on bounding_box or polygon input.\n",
    "\n",
    "if aoi == '1':\n",
    "# bounding box input:\n",
    "    params = {\n",
    "    'short_name': short_name,\n",
    "    'version': latest_version,\n",
    "    'temporal': temporal,\n",
    "    'page_size': 100,\n",
    "    'page_num': 1,\n",
    "    'bounding_box': bounding_box\n",
    "    }\n",
    "else:\n",
    "    \n",
    "# If polygon input (either via coordinate pairs or shapefile/KML/KMZ):\n",
    "    params = {\n",
    "    'short_name': short_name,\n",
    "    'version': latest_version,\n",
    "    'temporal': temporal,\n",
    "    'page_size': 100,\n",
    "    'page_num': 1,\n",
    "    'polygon': polygon,\n",
    "    }\n",
    "\n",
    "print('CMR search parameters: ', params)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Input the parameter dictionary to the CMR granule search to query all granules that meet the criteria based on the granule metadata. Print the number of granules returned."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "32"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Query number of granules using our (paging over results)\n",
    "\n",
    "granule_search_url = 'https://cmr.earthdata.nasa.gov/search/granules'\n",
    "\n",
    "granules = []\n",
    "while True:\n",
    "    response = requests.get(granule_search_url, params=params, headers=headers)\n",
    "    results = json.loads(response.content)\n",
    "\n",
    "    if len(results['feed']['entry']) == 0:\n",
    "        # Out of results, so break out of loop\n",
    "        break\n",
    "\n",
    "    # Collect results and increment page_num\n",
    "    granules.extend(results['feed']['entry'])\n",
    "    params['page_num'] += 1\n",
    "\n",
    "    \n",
    "# Get number of granules over my area and time of interest\n",
    "len(granules)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Now query the average size of those granules: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10.465339124196875\n",
      "334.8908519743\n"
     ]
    }
   ],
   "source": [
    "granule_sizes = [float(granule['granule_size']) for granule in granules]\n",
    "\n",
    "# Average size of granules in MB\n",
    "print(mean(granule_sizes))\n",
    "\n",
    "# Total volume in MB\n",
    "print(sum(granule_sizes))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Select the subsetting and reformatting services enabled for your data set of interest."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://n5eil02u.ecs.nsidc.org/egi/capabilities/ATL06.001.xml\n"
     ]
    }
   ],
   "source": [
    "# Query service capability URL \n",
    "\n",
    "from xml.etree import ElementTree as ET\n",
    "\n",
    "capability_url = f'https://n5eil02u.ecs.nsidc.org/egi/capabilities/{short_name}.{latest_version}.xml'\n",
    "\n",
    "print(capability_url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create session to store cookie and pass credentials to capabilities url\n",
    "\n",
    "session = requests.session()\n",
    "s = session.get(capability_url)\n",
    "response = session.get(s.url,auth=(uid,pswd))\n",
    "\n",
    "root = ET.fromstring(response.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the service capability XML, we can collect lists with each service option to gather service information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# collect lists with each service option\n",
    "\n",
    "subagent = [subset_agent.attrib for subset_agent in root.iter('SubsetAgent')]\n",
    "\n",
    "# variable subsetting\n",
    "variables = [SubsetVariable.attrib for SubsetVariable in root.iter('SubsetVariable')]  \n",
    "variables_raw = [variables[i]['value'] for i in range(len(variables))]\n",
    "variables_join = [''.join(('/',v)) if v.startswith('/') == False else v for v in variables_raw] \n",
    "variable_vals = [v.replace(':', '/') for v in variables_join]\n",
    "\n",
    "# reformatting\n",
    "formats = [Format.attrib for Format in root.iter('Format')]\n",
    "format_vals = [formats[i]['value'] for i in range(len(formats))]\n",
    "format_vals.remove('')\n",
    "\n",
    "# reprojection only applicable on ICESat-2 L3B products, yet to be available. \n",
    "\n",
    "# reformatting options that support reprojection\n",
    "normalproj = [Projections.attrib for Projections in root.iter('Projections')]\n",
    "normalproj_vals = []\n",
    "normalproj_vals.append(normalproj[0]['normalProj'])\n",
    "format_proj = normalproj_vals[0].split(',')\n",
    "format_proj.remove('')\n",
    "format_proj.append('No reformatting')\n",
    "\n",
    "#reprojection options\n",
    "projections = [Projection.attrib for Projection in root.iter('Projection')]\n",
    "proj_vals = []\n",
    "for i in range(len(projections)):\n",
    "    if (projections[i]['value']) != 'NO_CHANGE' :\n",
    "        proj_vals.append(projections[i]['value'])\n",
    "        \n",
    "# reformatting options that do not support reprojection\n",
    "no_proj = [i for i in format_vals if i not in format_proj]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Let's confirm that subset services exist for our data set by reviewing the `subagent` list. If the list contains service information, we know that services are available. If not, we need to set the `agent` API parameter to `NO` to indicate that our request will bypass the subsetter. This will quickly send back the data \"natively\" without any customization applied."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'id': 'ICESAT2', 'spatialSubsetting': 'true', 'spatialSubsettingShapefile': 'true', 'temporalSubsetting': 'true', 'type': 'both', 'maxGransSyncRequest': '100', 'maxGransAsyncRequest': '2000'}]\n"
     ]
    }
   ],
   "source": [
    "print(subagent)\n",
    "if len(subagent) < 1 :\n",
    "    agent = 'NO'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More information is contained in the subagent list, including the maximum number of granules that we can request per order depending on our configuration. We'll come back to these options below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### We'll begin populating the subsetting and reformatting parameters used for our NSIDC API request. In addition to the CMR information we queried above, the NSIDC API accepts Key-Value-Pairs (KVPs) for subsetting and reformatting services."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Let's start with spatial subsetting. Recall that there are three options to *filter* our search results by spatial constraint: \n",
    "\n",
    "1) Bounding Box: Corresponding to the CMR `bounding_box` KVP\n",
    "\n",
    "2) Polygon coordinate pairs: Corresponding to the CMR `polygon` KVP\n",
    "\n",
    "3) Spatial file input, including Esri Shapefile or KML/KMZ: We simplified the file input to also be read by the CMR `polygon` KVP \n",
    "    \n",
    "#### We see above that `spatialSubsetting` is `true` and `spatialSubsettingShapefile` is `true`. Therefore the same *filtering* options can be applied to our *subset* constraint, with unique KVPs for the subsetting service:\n",
    "\n",
    "1) Bounding Box: `bbox` subset KVP\n",
    "\n",
    "2) Polygon coordinate pairs: `bounding_shape` subset KVP in [GeoJSON](https://geojson.org/) format. \n",
    "\n",
    "3) Spatial file input: The file can be read directly by the subsetter without simplification. This file will be posted to the API endpoint, so we don't need to specify an additional subset KVP here. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Because we're pursuing option 3), we don't need to provide an additional subset parameter. Below is commented code for bounding box inputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Bounding box subsetting (bbox) in same format as bounding_box\n",
    "\n",
    "bbox = bounding_box\n",
    "\n",
    "#Polygon coordinate pair subsetting in GeoJSON format. Or for simplicity, get polygon bounds to be used as bounding box input\n",
    "\n",
    "# # Create shapely Polygon object from x y list\n",
    "# p = Polygon(tuple(xylist))\n",
    "# # Extract the point values that define the perimeter of the polygon\n",
    "# bounds = p.bounds\n",
    "# bbox = ','.join(map(str, list(bounds)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Temporal subsetting is next, since we saw above that `temporalSubsetting` is `true`. We filtered data over 22 Feb 2019 and we can also subset the data to those dates if desired. \n",
    "\n",
    "The `time` KVP is used to subset temporally. This can be entered in the following formats:\n",
    "\n",
    "`time=yyyy-mm-dd,yyyy-mm-dd`\n",
    "\n",
    "`time=yyy-mm-ddThh:MM:ss,yyy-mm-ddThh:MM:ss` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2018-12-12T00:00:00,2019-02-28T23:59:59\n"
     ]
    }
   ],
   "source": [
    "# Temporal subsetting KVP\n",
    "\n",
    "timevar = start_date + 'T' + start_time + ',' + end_date + 'T' + end_time\n",
    "print(timevar)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Finally, let's determine if variable subsetting is available by finding the length of the `variable_vals` list we gathered from the capabilities URL. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "627"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(variable_vals)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can view the entire list of variables if desired:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['/ancillary_data',\n",
      " '/ancillary_data/atlas_sdp_gps_epoch',\n",
      " '/ancillary_data/control',\n",
      " '/ancillary_data/data_end_utc',\n",
      " '/ancillary_data/data_start_utc',\n",
      " '/ancillary_data/end_cycle',\n",
      " '/ancillary_data/end_delta_time',\n",
      " '/ancillary_data/end_geoseg',\n",
      " '/ancillary_data/end_gpssow',\n",
      " '/ancillary_data/end_gpsweek',\n",
      " '/ancillary_data/end_orbit',\n",
      " '/ancillary_data/end_region',\n",
      " '/ancillary_data/end_rgt',\n",
      " '/ancillary_data/granule_end_utc',\n",
      " '/ancillary_data/granule_start_utc',\n",
      " '/ancillary_data/qa_at_interval',\n",
      " '/ancillary_data/release',\n",
      " '/ancillary_data/start_cycle',\n",
      " '/ancillary_data/start_delta_time',\n",
      " '/ancillary_data/start_geoseg',\n",
      " '/ancillary_data/start_gpssow',\n",
      " '/ancillary_data/start_gpsweek',\n",
      " '/ancillary_data/start_orbit',\n",
      " '/ancillary_data/start_region',\n",
      " '/ancillary_data/start_rgt',\n",
      " '/ancillary_data/version',\n",
      " '/ancillary_data/land_ice',\n",
      " '/ancillary_data/land_ice/dt_hist',\n",
      " '/ancillary_data/land_ice/fit_maxiter',\n",
      " '/ancillary_data/land_ice/fpb_maxiter',\n",
      " '/ancillary_data/land_ice/maxiter',\n",
      " '/ancillary_data/land_ice/max_res_ids',\n",
      " '/ancillary_data/land_ice/min_dist',\n",
      " '/ancillary_data/land_ice/min_gain_th',\n",
      " '/ancillary_data/land_ice/min_n_pe',\n",
      " '/ancillary_data/land_ice/min_n_sel',\n",
      " '/ancillary_data/land_ice/min_signal_conf',\n",
      " '/ancillary_data/land_ice/n_hist',\n",
      " '/ancillary_data/land_ice/nhist_bins',\n",
      " '/ancillary_data/land_ice/n_sigmas',\n",
      " '/ancillary_data/land_ice/proc_interval',\n",
      " '/ancillary_data/land_ice/rbin_width',\n",
      " '/ancillary_data/land_ice/sigma_beam',\n",
      " '/ancillary_data/land_ice/sigma_tx',\n",
      " '/ancillary_data/land_ice/t_dead',\n",
      " '/ancillary_data/land_ice/win_nsig',\n",
      " '/gt1l',\n",
      " '/gt1l/land_ice_segments',\n",
      " '/gt1l/land_ice_segments/atl06_quality_summary',\n",
      " '/gt1l/land_ice_segments/delta_time',\n",
      " '/gt1l/land_ice_segments/h_li',\n",
      " '/gt1l/land_ice_segments/h_li_sigma',\n",
      " '/gt1l/land_ice_segments/latitude',\n",
      " '/gt1l/land_ice_segments/longitude',\n",
      " '/gt1l/land_ice_segments/segment_id',\n",
      " '/gt1l/land_ice_segments/sigma_geo_h',\n",
      " '/gt1l/land_ice_segments/bias_correction',\n",
      " '/gt1l/land_ice_segments/bias_correction/fpb_mean_corr',\n",
      " '/gt1l/land_ice_segments/bias_correction/fpb_mean_corr_sigma',\n",
      " '/gt1l/land_ice_segments/bias_correction/fpb_med_corr',\n",
      " '/gt1l/land_ice_segments/bias_correction/fpb_med_corr_sigma',\n",
      " '/gt1l/land_ice_segments/bias_correction/fpb_n_corr',\n",
      " '/gt1l/land_ice_segments/bias_correction/med_r_fit',\n",
      " '/gt1l/land_ice_segments/bias_correction/tx_mean_corr',\n",
      " '/gt1l/land_ice_segments/bias_correction/tx_med_corr',\n",
      " '/gt1l/land_ice_segments/dem',\n",
      " '/gt1l/land_ice_segments/dem/dem_flag',\n",
      " '/gt1l/land_ice_segments/dem/dem_h',\n",
      " '/gt1l/land_ice_segments/dem/geoid_h',\n",
      " '/gt1l/land_ice_segments/fit_statistics',\n",
      " '/gt1l/land_ice_segments/fit_statistics/dh_fit_dx',\n",
      " '/gt1l/land_ice_segments/fit_statistics/dh_fit_dx_sigma',\n",
      " '/gt1l/land_ice_segments/fit_statistics/dh_fit_dy',\n",
      " '/gt1l/land_ice_segments/fit_statistics/h_expected_rms',\n",
      " '/gt1l/land_ice_segments/fit_statistics/h_mean',\n",
      " '/gt1l/land_ice_segments/fit_statistics/h_rms_misfit',\n",
      " '/gt1l/land_ice_segments/fit_statistics/h_robust_sprd',\n",
      " '/gt1l/land_ice_segments/fit_statistics/n_fit_photons',\n",
      " '/gt1l/land_ice_segments/fit_statistics/n_seg_pulses',\n",
      " '/gt1l/land_ice_segments/fit_statistics/sigma_h_mean',\n",
      " '/gt1l/land_ice_segments/fit_statistics/signal_selection_source',\n",
      " '/gt1l/land_ice_segments/fit_statistics/signal_selection_source_status',\n",
      " '/gt1l/land_ice_segments/fit_statistics/snr',\n",
      " '/gt1l/land_ice_segments/fit_statistics/snr_significance',\n",
      " '/gt1l/land_ice_segments/fit_statistics/w_surface_window_final',\n",
      " '/gt1l/land_ice_segments/geophysical',\n",
      " '/gt1l/land_ice_segments/geophysical/bckgrd',\n",
      " '/gt1l/land_ice_segments/geophysical/bsnow_conf',\n",
      " '/gt1l/land_ice_segments/geophysical/bsnow_h',\n",
      " '/gt1l/land_ice_segments/geophysical/bsnow_od',\n",
      " '/gt1l/land_ice_segments/geophysical/cloud_flg_asr',\n",
      " '/gt1l/land_ice_segments/geophysical/cloud_flg_atm',\n",
      " '/gt1l/land_ice_segments/geophysical/dac',\n",
      " '/gt1l/land_ice_segments/geophysical/e_bckgrd',\n",
      " '/gt1l/land_ice_segments/geophysical/msw_flag',\n",
      " '/gt1l/land_ice_segments/geophysical/neutat_delay_total',\n",
      " '/gt1l/land_ice_segments/geophysical/r_eff',\n",
      " '/gt1l/land_ice_segments/geophysical/solar_azimuth',\n",
      " '/gt1l/land_ice_segments/geophysical/solar_elevation',\n",
      " '/gt1l/land_ice_segments/geophysical/tide_earth',\n",
      " '/gt1l/land_ice_segments/geophysical/tide_load',\n",
      " '/gt1l/land_ice_segments/geophysical/tide_ocean',\n",
      " '/gt1l/land_ice_segments/geophysical/tide_pole',\n",
      " '/gt1l/land_ice_segments/ground_track',\n",
      " '/gt1l/land_ice_segments/ground_track/ref_azimuth',\n",
      " '/gt1l/land_ice_segments/ground_track/ref_coelv',\n",
      " '/gt1l/land_ice_segments/ground_track/seg_azimuth',\n",
      " '/gt1l/land_ice_segments/ground_track/sigma_geo_at',\n",
      " '/gt1l/land_ice_segments/ground_track/sigma_geo_xt',\n",
      " '/gt1l/land_ice_segments/ground_track/x_atc',\n",
      " '/gt1l/land_ice_segments/ground_track/y_atc',\n",
      " '/gt1l/residual_histogram',\n",
      " '/gt1l/residual_histogram/bckgrd_per_bin',\n",
      " '/gt1l/residual_histogram/count',\n",
      " '/gt1l/residual_histogram/delta_time',\n",
      " '/gt1l/residual_histogram/dh',\n",
      " '/gt1l/residual_histogram/ds_segment_id',\n",
      " '/gt1l/residual_histogram/lat_mean',\n",
      " '/gt1l/residual_histogram/lon_mean',\n",
      " '/gt1l/residual_histogram/pulse_count',\n",
      " '/gt1l/residual_histogram/segment_id_list',\n",
      " '/gt1l/residual_histogram/x_atc_mean',\n",
      " '/gt1l/segment_quality',\n",
      " '/gt1l/segment_quality/delta_time',\n",
      " '/gt1l/segment_quality/record_number',\n",
      " '/gt1l/segment_quality/reference_pt_lat',\n",
      " '/gt1l/segment_quality/reference_pt_lon',\n",
      " '/gt1l/segment_quality/segment_id',\n",
      " '/gt1l/segment_quality/signal_selection_source',\n",
      " '/gt1l/segment_quality/signal_selection_status',\n",
      " '/gt1l/segment_quality/signal_selection_status/signal_selection_status_all',\n",
      " '/gt1l/segment_quality/signal_selection_status/signal_selection_status_backup',\n",
      " '/gt1l/segment_quality/signal_selection_status/signal_selection_status_confident',\n",
      " '/gt1r',\n",
      " '/gt1r/land_ice_segments',\n",
      " '/gt1r/land_ice_segments/atl06_quality_summary',\n",
      " '/gt1r/land_ice_segments/delta_time',\n",
      " '/gt1r/land_ice_segments/h_li',\n",
      " '/gt1r/land_ice_segments/h_li_sigma',\n",
      " '/gt1r/land_ice_segments/latitude',\n",
      " '/gt1r/land_ice_segments/longitude',\n",
      " '/gt1r/land_ice_segments/segment_id',\n",
      " '/gt1r/land_ice_segments/sigma_geo_h',\n",
      " '/gt1r/land_ice_segments/bias_correction',\n",
      " '/gt1r/land_ice_segments/bias_correction/fpb_mean_corr',\n",
      " '/gt1r/land_ice_segments/bias_correction/fpb_mean_corr_sigma',\n",
      " '/gt1r/land_ice_segments/bias_correction/fpb_med_corr',\n",
      " '/gt1r/land_ice_segments/bias_correction/fpb_med_corr_sigma',\n",
      " '/gt1r/land_ice_segments/bias_correction/fpb_n_corr',\n",
      " '/gt1r/land_ice_segments/bias_correction/med_r_fit',\n",
      " '/gt1r/land_ice_segments/bias_correction/tx_mean_corr',\n",
      " '/gt1r/land_ice_segments/bias_correction/tx_med_corr',\n",
      " '/gt1r/land_ice_segments/dem',\n",
      " '/gt1r/land_ice_segments/dem/dem_flag',\n",
      " '/gt1r/land_ice_segments/dem/dem_h',\n",
      " '/gt1r/land_ice_segments/dem/geoid_h',\n",
      " '/gt1r/land_ice_segments/fit_statistics',\n",
      " '/gt1r/land_ice_segments/fit_statistics/dh_fit_dx',\n",
      " '/gt1r/land_ice_segments/fit_statistics/dh_fit_dx_sigma',\n",
      " '/gt1r/land_ice_segments/fit_statistics/dh_fit_dy',\n",
      " '/gt1r/land_ice_segments/fit_statistics/h_expected_rms',\n",
      " '/gt1r/land_ice_segments/fit_statistics/h_mean',\n",
      " '/gt1r/land_ice_segments/fit_statistics/h_rms_misfit',\n",
      " '/gt1r/land_ice_segments/fit_statistics/h_robust_sprd',\n",
      " '/gt1r/land_ice_segments/fit_statistics/n_fit_photons',\n",
      " '/gt1r/land_ice_segments/fit_statistics/n_seg_pulses',\n",
      " '/gt1r/land_ice_segments/fit_statistics/sigma_h_mean',\n",
      " '/gt1r/land_ice_segments/fit_statistics/signal_selection_source',\n",
      " '/gt1r/land_ice_segments/fit_statistics/signal_selection_source_status',\n",
      " '/gt1r/land_ice_segments/fit_statistics/snr',\n",
      " '/gt1r/land_ice_segments/fit_statistics/snr_significance',\n",
      " '/gt1r/land_ice_segments/fit_statistics/w_surface_window_final',\n",
      " '/gt1r/land_ice_segments/geophysical',\n",
      " '/gt1r/land_ice_segments/geophysical/bckgrd',\n",
      " '/gt1r/land_ice_segments/geophysical/bsnow_conf',\n",
      " '/gt1r/land_ice_segments/geophysical/bsnow_h',\n",
      " '/gt1r/land_ice_segments/geophysical/bsnow_od',\n",
      " '/gt1r/land_ice_segments/geophysical/cloud_flg_asr',\n",
      " '/gt1r/land_ice_segments/geophysical/cloud_flg_atm',\n",
      " '/gt1r/land_ice_segments/geophysical/dac',\n",
      " '/gt1r/land_ice_segments/geophysical/e_bckgrd',\n",
      " '/gt1r/land_ice_segments/geophysical/msw_flag',\n",
      " '/gt1r/land_ice_segments/geophysical/neutat_delay_total',\n",
      " '/gt1r/land_ice_segments/geophysical/r_eff',\n",
      " '/gt1r/land_ice_segments/geophysical/solar_azimuth',\n",
      " '/gt1r/land_ice_segments/geophysical/solar_elevation',\n",
      " '/gt1r/land_ice_segments/geophysical/tide_earth',\n",
      " '/gt1r/land_ice_segments/geophysical/tide_load',\n",
      " '/gt1r/land_ice_segments/geophysical/tide_ocean',\n",
      " '/gt1r/land_ice_segments/geophysical/tide_pole',\n",
      " '/gt1r/land_ice_segments/ground_track',\n",
      " '/gt1r/land_ice_segments/ground_track/ref_azimuth',\n",
      " '/gt1r/land_ice_segments/ground_track/ref_coelv',\n",
      " '/gt1r/land_ice_segments/ground_track/seg_azimuth',\n",
      " '/gt1r/land_ice_segments/ground_track/sigma_geo_at',\n",
      " '/gt1r/land_ice_segments/ground_track/sigma_geo_xt',\n",
      " '/gt1r/land_ice_segments/ground_track/x_atc',\n",
      " '/gt1r/land_ice_segments/ground_track/y_atc',\n",
      " '/gt1r/residual_histogram',\n",
      " '/gt1r/residual_histogram/bckgrd_per_bin',\n",
      " '/gt1r/residual_histogram/count',\n",
      " '/gt1r/residual_histogram/delta_time',\n",
      " '/gt1r/residual_histogram/dh',\n",
      " '/gt1r/residual_histogram/ds_segment_id',\n",
      " '/gt1r/residual_histogram/lat_mean',\n",
      " '/gt1r/residual_histogram/lon_mean',\n",
      " '/gt1r/residual_histogram/pulse_count',\n",
      " '/gt1r/residual_histogram/segment_id_list',\n",
      " '/gt1r/residual_histogram/x_atc_mean',\n",
      " '/gt1r/segment_quality',\n",
      " '/gt1r/segment_quality/delta_time',\n",
      " '/gt1r/segment_quality/record_number',\n",
      " '/gt1r/segment_quality/reference_pt_lat',\n",
      " '/gt1r/segment_quality/reference_pt_lon',\n",
      " '/gt1r/segment_quality/segment_id',\n",
      " '/gt1r/segment_quality/signal_selection_source',\n",
      " '/gt1r/segment_quality/signal_selection_status',\n",
      " '/gt1r/segment_quality/signal_selection_status/signal_selection_status_all',\n",
      " '/gt1r/segment_quality/signal_selection_status/signal_selection_status_backup',\n",
      " '/gt1r/segment_quality/signal_selection_status/signal_selection_status_confident',\n",
      " '/gt2l',\n",
      " '/gt2l/land_ice_segments',\n",
      " '/gt2l/land_ice_segments/atl06_quality_summary',\n",
      " '/gt2l/land_ice_segments/delta_time',\n",
      " '/gt2l/land_ice_segments/h_li',\n",
      " '/gt2l/land_ice_segments/h_li_sigma',\n",
      " '/gt2l/land_ice_segments/latitude',\n",
      " '/gt2l/land_ice_segments/longitude',\n",
      " '/gt2l/land_ice_segments/segment_id',\n",
      " '/gt2l/land_ice_segments/sigma_geo_h',\n",
      " '/gt2l/land_ice_segments/bias_correction',\n",
      " '/gt2l/land_ice_segments/bias_correction/fpb_mean_corr',\n",
      " '/gt2l/land_ice_segments/bias_correction/fpb_mean_corr_sigma',\n",
      " '/gt2l/land_ice_segments/bias_correction/fpb_med_corr',\n",
      " '/gt2l/land_ice_segments/bias_correction/fpb_med_corr_sigma',\n",
      " '/gt2l/land_ice_segments/bias_correction/fpb_n_corr',\n",
      " '/gt2l/land_ice_segments/bias_correction/med_r_fit',\n",
      " '/gt2l/land_ice_segments/bias_correction/tx_mean_corr',\n",
      " '/gt2l/land_ice_segments/bias_correction/tx_med_corr',\n",
      " '/gt2l/land_ice_segments/dem',\n",
      " '/gt2l/land_ice_segments/dem/dem_flag',\n",
      " '/gt2l/land_ice_segments/dem/dem_h',\n",
      " '/gt2l/land_ice_segments/dem/geoid_h',\n",
      " '/gt2l/land_ice_segments/fit_statistics',\n",
      " '/gt2l/land_ice_segments/fit_statistics/dh_fit_dx',\n",
      " '/gt2l/land_ice_segments/fit_statistics/dh_fit_dx_sigma',\n",
      " '/gt2l/land_ice_segments/fit_statistics/dh_fit_dy',\n",
      " '/gt2l/land_ice_segments/fit_statistics/h_expected_rms',\n",
      " '/gt2l/land_ice_segments/fit_statistics/h_mean',\n",
      " '/gt2l/land_ice_segments/fit_statistics/h_rms_misfit',\n",
      " '/gt2l/land_ice_segments/fit_statistics/h_robust_sprd',\n",
      " '/gt2l/land_ice_segments/fit_statistics/n_fit_photons',\n",
      " '/gt2l/land_ice_segments/fit_statistics/n_seg_pulses',\n",
      " '/gt2l/land_ice_segments/fit_statistics/sigma_h_mean',\n",
      " '/gt2l/land_ice_segments/fit_statistics/signal_selection_source',\n",
      " '/gt2l/land_ice_segments/fit_statistics/signal_selection_source_status',\n",
      " '/gt2l/land_ice_segments/fit_statistics/snr',\n",
      " '/gt2l/land_ice_segments/fit_statistics/snr_significance',\n",
      " '/gt2l/land_ice_segments/fit_statistics/w_surface_window_final',\n",
      " '/gt2l/land_ice_segments/geophysical',\n",
      " '/gt2l/land_ice_segments/geophysical/bckgrd',\n",
      " '/gt2l/land_ice_segments/geophysical/bsnow_conf',\n",
      " '/gt2l/land_ice_segments/geophysical/bsnow_h',\n",
      " '/gt2l/land_ice_segments/geophysical/bsnow_od',\n",
      " '/gt2l/land_ice_segments/geophysical/cloud_flg_asr',\n",
      " '/gt2l/land_ice_segments/geophysical/cloud_flg_atm',\n",
      " '/gt2l/land_ice_segments/geophysical/dac',\n",
      " '/gt2l/land_ice_segments/geophysical/e_bckgrd',\n",
      " '/gt2l/land_ice_segments/geophysical/msw_flag',\n",
      " '/gt2l/land_ice_segments/geophysical/neutat_delay_total',\n",
      " '/gt2l/land_ice_segments/geophysical/r_eff',\n",
      " '/gt2l/land_ice_segments/geophysical/solar_azimuth',\n",
      " '/gt2l/land_ice_segments/geophysical/solar_elevation',\n",
      " '/gt2l/land_ice_segments/geophysical/tide_earth',\n",
      " '/gt2l/land_ice_segments/geophysical/tide_load',\n",
      " '/gt2l/land_ice_segments/geophysical/tide_ocean',\n",
      " '/gt2l/land_ice_segments/geophysical/tide_pole',\n",
      " '/gt2l/land_ice_segments/ground_track',\n",
      " '/gt2l/land_ice_segments/ground_track/ref_azimuth',\n",
      " '/gt2l/land_ice_segments/ground_track/ref_coelv',\n",
      " '/gt2l/land_ice_segments/ground_track/seg_azimuth',\n",
      " '/gt2l/land_ice_segments/ground_track/sigma_geo_at',\n",
      " '/gt2l/land_ice_segments/ground_track/sigma_geo_xt',\n",
      " '/gt2l/land_ice_segments/ground_track/x_atc',\n",
      " '/gt2l/land_ice_segments/ground_track/y_atc',\n",
      " '/gt2l/residual_histogram',\n",
      " '/gt2l/residual_histogram/bckgrd_per_bin',\n",
      " '/gt2l/residual_histogram/count',\n",
      " '/gt2l/residual_histogram/delta_time',\n",
      " '/gt2l/residual_histogram/dh',\n",
      " '/gt2l/residual_histogram/ds_segment_id',\n",
      " '/gt2l/residual_histogram/lat_mean',\n",
      " '/gt2l/residual_histogram/lon_mean',\n",
      " '/gt2l/residual_histogram/pulse_count',\n",
      " '/gt2l/residual_histogram/segment_id_list',\n",
      " '/gt2l/residual_histogram/x_atc_mean',\n",
      " '/gt2l/segment_quality',\n",
      " '/gt2l/segment_quality/delta_time',\n",
      " '/gt2l/segment_quality/record_number',\n",
      " '/gt2l/segment_quality/reference_pt_lat',\n",
      " '/gt2l/segment_quality/reference_pt_lon',\n",
      " '/gt2l/segment_quality/segment_id',\n",
      " '/gt2l/segment_quality/signal_selection_source',\n",
      " '/gt2l/segment_quality/signal_selection_status',\n",
      " '/gt2l/segment_quality/signal_selection_status/signal_selection_status_all',\n",
      " '/gt2l/segment_quality/signal_selection_status/signal_selection_status_backup',\n",
      " '/gt2l/segment_quality/signal_selection_status/signal_selection_status_confident',\n",
      " '/gt2r',\n",
      " '/gt2r/land_ice_segments',\n",
      " '/gt2r/land_ice_segments/atl06_quality_summary',\n",
      " '/gt2r/land_ice_segments/delta_time',\n",
      " '/gt2r/land_ice_segments/h_li',\n",
      " '/gt2r/land_ice_segments/h_li_sigma',\n",
      " '/gt2r/land_ice_segments/latitude',\n",
      " '/gt2r/land_ice_segments/longitude',\n",
      " '/gt2r/land_ice_segments/segment_id',\n",
      " '/gt2r/land_ice_segments/sigma_geo_h',\n",
      " '/gt2r/land_ice_segments/bias_correction',\n",
      " '/gt2r/land_ice_segments/bias_correction/fpb_mean_corr',\n",
      " '/gt2r/land_ice_segments/bias_correction/fpb_mean_corr_sigma',\n",
      " '/gt2r/land_ice_segments/bias_correction/fpb_med_corr',\n",
      " '/gt2r/land_ice_segments/bias_correction/fpb_med_corr_sigma',\n",
      " '/gt2r/land_ice_segments/bias_correction/fpb_n_corr',\n",
      " '/gt2r/land_ice_segments/bias_correction/med_r_fit',\n",
      " '/gt2r/land_ice_segments/bias_correction/tx_mean_corr',\n",
      " '/gt2r/land_ice_segments/bias_correction/tx_med_corr',\n",
      " '/gt2r/land_ice_segments/dem',\n",
      " '/gt2r/land_ice_segments/dem/dem_flag',\n",
      " '/gt2r/land_ice_segments/dem/dem_h',\n",
      " '/gt2r/land_ice_segments/dem/geoid_h',\n",
      " '/gt2r/land_ice_segments/fit_statistics',\n",
      " '/gt2r/land_ice_segments/fit_statistics/dh_fit_dx',\n",
      " '/gt2r/land_ice_segments/fit_statistics/dh_fit_dx_sigma',\n",
      " '/gt2r/land_ice_segments/fit_statistics/dh_fit_dy',\n",
      " '/gt2r/land_ice_segments/fit_statistics/h_expected_rms',\n",
      " '/gt2r/land_ice_segments/fit_statistics/h_mean',\n",
      " '/gt2r/land_ice_segments/fit_statistics/h_rms_misfit',\n",
      " '/gt2r/land_ice_segments/fit_statistics/h_robust_sprd',\n",
      " '/gt2r/land_ice_segments/fit_statistics/n_fit_photons',\n",
      " '/gt2r/land_ice_segments/fit_statistics/n_seg_pulses',\n",
      " '/gt2r/land_ice_segments/fit_statistics/sigma_h_mean',\n",
      " '/gt2r/land_ice_segments/fit_statistics/signal_selection_source',\n",
      " '/gt2r/land_ice_segments/fit_statistics/signal_selection_source_status',\n",
      " '/gt2r/land_ice_segments/fit_statistics/snr',\n",
      " '/gt2r/land_ice_segments/fit_statistics/snr_significance',\n",
      " '/gt2r/land_ice_segments/fit_statistics/w_surface_window_final',\n",
      " '/gt2r/land_ice_segments/geophysical',\n",
      " '/gt2r/land_ice_segments/geophysical/bckgrd',\n",
      " '/gt2r/land_ice_segments/geophysical/bsnow_conf',\n",
      " '/gt2r/land_ice_segments/geophysical/bsnow_h',\n",
      " '/gt2r/land_ice_segments/geophysical/bsnow_od',\n",
      " '/gt2r/land_ice_segments/geophysical/cloud_flg_asr',\n",
      " '/gt2r/land_ice_segments/geophysical/cloud_flg_atm',\n",
      " '/gt2r/land_ice_segments/geophysical/dac',\n",
      " '/gt2r/land_ice_segments/geophysical/e_bckgrd',\n",
      " '/gt2r/land_ice_segments/geophysical/msw_flag',\n",
      " '/gt2r/land_ice_segments/geophysical/neutat_delay_total',\n",
      " '/gt2r/land_ice_segments/geophysical/r_eff',\n",
      " '/gt2r/land_ice_segments/geophysical/solar_azimuth',\n",
      " '/gt2r/land_ice_segments/geophysical/solar_elevation',\n",
      " '/gt2r/land_ice_segments/geophysical/tide_earth',\n",
      " '/gt2r/land_ice_segments/geophysical/tide_load',\n",
      " '/gt2r/land_ice_segments/geophysical/tide_ocean',\n",
      " '/gt2r/land_ice_segments/geophysical/tide_pole',\n",
      " '/gt2r/land_ice_segments/ground_track',\n",
      " '/gt2r/land_ice_segments/ground_track/ref_azimuth',\n",
      " '/gt2r/land_ice_segments/ground_track/ref_coelv',\n",
      " '/gt2r/land_ice_segments/ground_track/seg_azimuth',\n",
      " '/gt2r/land_ice_segments/ground_track/sigma_geo_at',\n",
      " '/gt2r/land_ice_segments/ground_track/sigma_geo_xt',\n",
      " '/gt2r/land_ice_segments/ground_track/x_atc',\n",
      " '/gt2r/land_ice_segments/ground_track/y_atc',\n",
      " '/gt2r/residual_histogram',\n",
      " '/gt2r/residual_histogram/bckgrd_per_bin',\n",
      " '/gt2r/residual_histogram/count',\n",
      " '/gt2r/residual_histogram/delta_time',\n",
      " '/gt2r/residual_histogram/dh',\n",
      " '/gt2r/residual_histogram/ds_segment_id',\n",
      " '/gt2r/residual_histogram/lat_mean',\n",
      " '/gt2r/residual_histogram/lon_mean',\n",
      " '/gt2r/residual_histogram/pulse_count',\n",
      " '/gt2r/residual_histogram/segment_id_list',\n",
      " '/gt2r/residual_histogram/x_atc_mean',\n",
      " '/gt2r/segment_quality',\n",
      " '/gt2r/segment_quality/delta_time',\n",
      " '/gt2r/segment_quality/record_number',\n",
      " '/gt2r/segment_quality/reference_pt_lat',\n",
      " '/gt2r/segment_quality/reference_pt_lon',\n",
      " '/gt2r/segment_quality/segment_id',\n",
      " '/gt2r/segment_quality/signal_selection_source',\n",
      " '/gt2r/segment_quality/signal_selection_status',\n",
      " '/gt2r/segment_quality/signal_selection_status/signal_selection_status_all',\n",
      " '/gt2r/segment_quality/signal_selection_status/signal_selection_status_backup',\n",
      " '/gt2r/segment_quality/signal_selection_status/signal_selection_status_confident',\n",
      " '/gt3l',\n",
      " '/gt3l/land_ice_segments',\n",
      " '/gt3l/land_ice_segments/atl06_quality_summary',\n",
      " '/gt3l/land_ice_segments/delta_time',\n",
      " '/gt3l/land_ice_segments/h_li',\n",
      " '/gt3l/land_ice_segments/h_li_sigma',\n",
      " '/gt3l/land_ice_segments/latitude',\n",
      " '/gt3l/land_ice_segments/longitude',\n",
      " '/gt3l/land_ice_segments/segment_id',\n",
      " '/gt3l/land_ice_segments/sigma_geo_h',\n",
      " '/gt3l/land_ice_segments/bias_correction',\n",
      " '/gt3l/land_ice_segments/bias_correction/fpb_mean_corr',\n",
      " '/gt3l/land_ice_segments/bias_correction/fpb_mean_corr_sigma',\n",
      " '/gt3l/land_ice_segments/bias_correction/fpb_med_corr',\n",
      " '/gt3l/land_ice_segments/bias_correction/fpb_med_corr_sigma',\n",
      " '/gt3l/land_ice_segments/bias_correction/fpb_n_corr',\n",
      " '/gt3l/land_ice_segments/bias_correction/med_r_fit',\n",
      " '/gt3l/land_ice_segments/bias_correction/tx_mean_corr',\n",
      " '/gt3l/land_ice_segments/bias_correction/tx_med_corr',\n",
      " '/gt3l/land_ice_segments/dem',\n",
      " '/gt3l/land_ice_segments/dem/dem_flag',\n",
      " '/gt3l/land_ice_segments/dem/dem_h',\n",
      " '/gt3l/land_ice_segments/dem/geoid_h',\n",
      " '/gt3l/land_ice_segments/fit_statistics',\n",
      " '/gt3l/land_ice_segments/fit_statistics/dh_fit_dx',\n",
      " '/gt3l/land_ice_segments/fit_statistics/dh_fit_dx_sigma',\n",
      " '/gt3l/land_ice_segments/fit_statistics/dh_fit_dy',\n",
      " '/gt3l/land_ice_segments/fit_statistics/h_expected_rms',\n",
      " '/gt3l/land_ice_segments/fit_statistics/h_mean',\n",
      " '/gt3l/land_ice_segments/fit_statistics/h_rms_misfit',\n",
      " '/gt3l/land_ice_segments/fit_statistics/h_robust_sprd',\n",
      " '/gt3l/land_ice_segments/fit_statistics/n_fit_photons',\n",
      " '/gt3l/land_ice_segments/fit_statistics/n_seg_pulses',\n",
      " '/gt3l/land_ice_segments/fit_statistics/sigma_h_mean',\n",
      " '/gt3l/land_ice_segments/fit_statistics/signal_selection_source',\n",
      " '/gt3l/land_ice_segments/fit_statistics/signal_selection_source_status',\n",
      " '/gt3l/land_ice_segments/fit_statistics/snr',\n",
      " '/gt3l/land_ice_segments/fit_statistics/snr_significance',\n",
      " '/gt3l/land_ice_segments/fit_statistics/w_surface_window_final',\n",
      " '/gt3l/land_ice_segments/geophysical',\n",
      " '/gt3l/land_ice_segments/geophysical/bckgrd',\n",
      " '/gt3l/land_ice_segments/geophysical/bsnow_conf',\n",
      " '/gt3l/land_ice_segments/geophysical/bsnow_h',\n",
      " '/gt3l/land_ice_segments/geophysical/bsnow_od',\n",
      " '/gt3l/land_ice_segments/geophysical/cloud_flg_asr',\n",
      " '/gt3l/land_ice_segments/geophysical/cloud_flg_atm',\n",
      " '/gt3l/land_ice_segments/geophysical/dac',\n",
      " '/gt3l/land_ice_segments/geophysical/e_bckgrd',\n",
      " '/gt3l/land_ice_segments/geophysical/msw_flag',\n",
      " '/gt3l/land_ice_segments/geophysical/neutat_delay_total',\n",
      " '/gt3l/land_ice_segments/geophysical/r_eff',\n",
      " '/gt3l/land_ice_segments/geophysical/solar_azimuth',\n",
      " '/gt3l/land_ice_segments/geophysical/solar_elevation',\n",
      " '/gt3l/land_ice_segments/geophysical/tide_earth',\n",
      " '/gt3l/land_ice_segments/geophysical/tide_load',\n",
      " '/gt3l/land_ice_segments/geophysical/tide_ocean',\n",
      " '/gt3l/land_ice_segments/geophysical/tide_pole',\n",
      " '/gt3l/land_ice_segments/ground_track',\n",
      " '/gt3l/land_ice_segments/ground_track/ref_azimuth',\n",
      " '/gt3l/land_ice_segments/ground_track/ref_coelv',\n",
      " '/gt3l/land_ice_segments/ground_track/seg_azimuth',\n",
      " '/gt3l/land_ice_segments/ground_track/sigma_geo_at',\n",
      " '/gt3l/land_ice_segments/ground_track/sigma_geo_xt',\n",
      " '/gt3l/land_ice_segments/ground_track/x_atc',\n",
      " '/gt3l/land_ice_segments/ground_track/y_atc',\n",
      " '/gt3l/residual_histogram',\n",
      " '/gt3l/residual_histogram/bckgrd_per_bin',\n",
      " '/gt3l/residual_histogram/count',\n",
      " '/gt3l/residual_histogram/delta_time',\n",
      " '/gt3l/residual_histogram/dh',\n",
      " '/gt3l/residual_histogram/ds_segment_id',\n",
      " '/gt3l/residual_histogram/lat_mean',\n",
      " '/gt3l/residual_histogram/lon_mean',\n",
      " '/gt3l/residual_histogram/pulse_count',\n",
      " '/gt3l/residual_histogram/segment_id_list',\n",
      " '/gt3l/residual_histogram/x_atc_mean',\n",
      " '/gt3l/segment_quality',\n",
      " '/gt3l/segment_quality/delta_time',\n",
      " '/gt3l/segment_quality/record_number',\n",
      " '/gt3l/segment_quality/reference_pt_lat',\n",
      " '/gt3l/segment_quality/reference_pt_lon',\n",
      " '/gt3l/segment_quality/segment_id',\n",
      " '/gt3l/segment_quality/signal_selection_source',\n",
      " '/gt3l/segment_quality/signal_selection_status',\n",
      " '/gt3l/segment_quality/signal_selection_status/signal_selection_status_all',\n",
      " '/gt3l/segment_quality/signal_selection_status/signal_selection_status_backup',\n",
      " '/gt3l/segment_quality/signal_selection_status/signal_selection_status_confident',\n",
      " '/gt3r',\n",
      " '/gt3r/land_ice_segments',\n",
      " '/gt3r/land_ice_segments/atl06_quality_summary',\n",
      " '/gt3r/land_ice_segments/delta_time',\n",
      " '/gt3r/land_ice_segments/h_li',\n",
      " '/gt3r/land_ice_segments/h_li_sigma',\n",
      " '/gt3r/land_ice_segments/latitude',\n",
      " '/gt3r/land_ice_segments/longitude',\n",
      " '/gt3r/land_ice_segments/segment_id',\n",
      " '/gt3r/land_ice_segments/sigma_geo_h',\n",
      " '/gt3r/land_ice_segments/bias_correction',\n",
      " '/gt3r/land_ice_segments/bias_correction/fpb_mean_corr',\n",
      " '/gt3r/land_ice_segments/bias_correction/fpb_mean_corr_sigma',\n",
      " '/gt3r/land_ice_segments/bias_correction/fpb_med_corr',\n",
      " '/gt3r/land_ice_segments/bias_correction/fpb_med_corr_sigma',\n",
      " '/gt3r/land_ice_segments/bias_correction/fpb_n_corr',\n",
      " '/gt3r/land_ice_segments/bias_correction/med_r_fit',\n",
      " '/gt3r/land_ice_segments/bias_correction/tx_mean_corr',\n",
      " '/gt3r/land_ice_segments/bias_correction/tx_med_corr',\n",
      " '/gt3r/land_ice_segments/dem',\n",
      " '/gt3r/land_ice_segments/dem/dem_flag',\n",
      " '/gt3r/land_ice_segments/dem/dem_h',\n",
      " '/gt3r/land_ice_segments/dem/geoid_h',\n",
      " '/gt3r/land_ice_segments/fit_statistics',\n",
      " '/gt3r/land_ice_segments/fit_statistics/dh_fit_dx',\n",
      " '/gt3r/land_ice_segments/fit_statistics/dh_fit_dx_sigma',\n",
      " '/gt3r/land_ice_segments/fit_statistics/dh_fit_dy',\n",
      " '/gt3r/land_ice_segments/fit_statistics/h_expected_rms',\n",
      " '/gt3r/land_ice_segments/fit_statistics/h_mean',\n",
      " '/gt3r/land_ice_segments/fit_statistics/h_rms_misfit',\n",
      " '/gt3r/land_ice_segments/fit_statistics/h_robust_sprd',\n",
      " '/gt3r/land_ice_segments/fit_statistics/n_fit_photons',\n",
      " '/gt3r/land_ice_segments/fit_statistics/n_seg_pulses',\n",
      " '/gt3r/land_ice_segments/fit_statistics/sigma_h_mean',\n",
      " '/gt3r/land_ice_segments/fit_statistics/signal_selection_source',\n",
      " '/gt3r/land_ice_segments/fit_statistics/signal_selection_source_status',\n",
      " '/gt3r/land_ice_segments/fit_statistics/snr',\n",
      " '/gt3r/land_ice_segments/fit_statistics/snr_significance',\n",
      " '/gt3r/land_ice_segments/fit_statistics/w_surface_window_final',\n",
      " '/gt3r/land_ice_segments/geophysical',\n",
      " '/gt3r/land_ice_segments/geophysical/bckgrd',\n",
      " '/gt3r/land_ice_segments/geophysical/bsnow_conf',\n",
      " '/gt3r/land_ice_segments/geophysical/bsnow_h',\n",
      " '/gt3r/land_ice_segments/geophysical/bsnow_od',\n",
      " '/gt3r/land_ice_segments/geophysical/cloud_flg_asr',\n",
      " '/gt3r/land_ice_segments/geophysical/cloud_flg_atm',\n",
      " '/gt3r/land_ice_segments/geophysical/dac',\n",
      " '/gt3r/land_ice_segments/geophysical/e_bckgrd',\n",
      " '/gt3r/land_ice_segments/geophysical/msw_flag',\n",
      " '/gt3r/land_ice_segments/geophysical/neutat_delay_total',\n",
      " '/gt3r/land_ice_segments/geophysical/r_eff',\n",
      " '/gt3r/land_ice_segments/geophysical/solar_azimuth',\n",
      " '/gt3r/land_ice_segments/geophysical/solar_elevation',\n",
      " '/gt3r/land_ice_segments/geophysical/tide_earth',\n",
      " '/gt3r/land_ice_segments/geophysical/tide_load',\n",
      " '/gt3r/land_ice_segments/geophysical/tide_ocean',\n",
      " '/gt3r/land_ice_segments/geophysical/tide_pole',\n",
      " '/gt3r/land_ice_segments/ground_track',\n",
      " '/gt3r/land_ice_segments/ground_track/ref_azimuth',\n",
      " '/gt3r/land_ice_segments/ground_track/ref_coelv',\n",
      " '/gt3r/land_ice_segments/ground_track/seg_azimuth',\n",
      " '/gt3r/land_ice_segments/ground_track/sigma_geo_at',\n",
      " '/gt3r/land_ice_segments/ground_track/sigma_geo_xt',\n",
      " '/gt3r/land_ice_segments/ground_track/x_atc',\n",
      " '/gt3r/land_ice_segments/ground_track/y_atc',\n",
      " '/gt3r/residual_histogram',\n",
      " '/gt3r/residual_histogram/bckgrd_per_bin',\n",
      " '/gt3r/residual_histogram/count',\n",
      " '/gt3r/residual_histogram/delta_time',\n",
      " '/gt3r/residual_histogram/dh',\n",
      " '/gt3r/residual_histogram/ds_segment_id',\n",
      " '/gt3r/residual_histogram/lat_mean',\n",
      " '/gt3r/residual_histogram/lon_mean',\n",
      " '/gt3r/residual_histogram/pulse_count',\n",
      " '/gt3r/residual_histogram/segment_id_list',\n",
      " '/gt3r/residual_histogram/x_atc_mean',\n",
      " '/gt3r/segment_quality',\n",
      " '/gt3r/segment_quality/delta_time',\n",
      " '/gt3r/segment_quality/record_number',\n",
      " '/gt3r/segment_quality/reference_pt_lat',\n",
      " '/gt3r/segment_quality/reference_pt_lon',\n",
      " '/gt3r/segment_quality/segment_id',\n",
      " '/gt3r/segment_quality/signal_selection_source',\n",
      " '/gt3r/segment_quality/signal_selection_status',\n",
      " '/gt3r/segment_quality/signal_selection_status/signal_selection_status_all',\n",
      " '/gt3r/segment_quality/signal_selection_status/signal_selection_status_backup',\n",
      " '/gt3r/segment_quality/signal_selection_status/signal_selection_status_confident',\n",
      " '/orbit_info',\n",
      " '/orbit_info/crossing_time',\n",
      " '/orbit_info/cycle_number',\n",
      " '/orbit_info/lan',\n",
      " '/orbit_info/orbit_number',\n",
      " '/orbit_info/rgt',\n",
      " '/orbit_info/sc_orient',\n",
      " '/orbit_info/sc_orient_time',\n",
      " '/quality_assessment',\n",
      " '/quality_assessment/qa_granule_fail_reason',\n",
      " '/quality_assessment/qa_granule_pass_fail',\n",
      " '/quality_assessment/gt1l',\n",
      " '/quality_assessment/gt1l/delta_time',\n",
      " '/quality_assessment/gt1l/lat_mean',\n",
      " '/quality_assessment/gt1l/lon_mean',\n",
      " '/quality_assessment/gt1l/signal_selection_source_fraction_0',\n",
      " '/quality_assessment/gt1l/signal_selection_source_fraction_1',\n",
      " '/quality_assessment/gt1l/signal_selection_source_fraction_2',\n",
      " '/quality_assessment/gt1l/signal_selection_source_fraction_3',\n",
      " '/quality_assessment/gt1r',\n",
      " '/quality_assessment/gt1r/delta_time',\n",
      " '/quality_assessment/gt1r/lat_mean',\n",
      " '/quality_assessment/gt1r/lon_mean',\n",
      " '/quality_assessment/gt1r/signal_selection_source_fraction_0',\n",
      " '/quality_assessment/gt1r/signal_selection_source_fraction_1',\n",
      " '/quality_assessment/gt1r/signal_selection_source_fraction_2',\n",
      " '/quality_assessment/gt1r/signal_selection_source_fraction_3',\n",
      " '/quality_assessment/gt2l',\n",
      " '/quality_assessment/gt2l/delta_time',\n",
      " '/quality_assessment/gt2l/lat_mean',\n",
      " '/quality_assessment/gt2l/lon_mean',\n",
      " '/quality_assessment/gt2l/signal_selection_source_fraction_0',\n",
      " '/quality_assessment/gt2l/signal_selection_source_fraction_1',\n",
      " '/quality_assessment/gt2l/signal_selection_source_fraction_2',\n",
      " '/quality_assessment/gt2l/signal_selection_source_fraction_3',\n",
      " '/quality_assessment/gt2r',\n",
      " '/quality_assessment/gt2r/delta_time',\n",
      " '/quality_assessment/gt2r/lat_mean',\n",
      " '/quality_assessment/gt2r/lon_mean',\n",
      " '/quality_assessment/gt2r/signal_selection_source_fraction_0',\n",
      " '/quality_assessment/gt2r/signal_selection_source_fraction_1',\n",
      " '/quality_assessment/gt2r/signal_selection_source_fraction_2',\n",
      " '/quality_assessment/gt2r/signal_selection_source_fraction_3',\n",
      " '/quality_assessment/gt3l',\n",
      " '/quality_assessment/gt3l/delta_time',\n",
      " '/quality_assessment/gt3l/lat_mean',\n",
      " '/quality_assessment/gt3l/lon_mean',\n",
      " '/quality_assessment/gt3l/signal_selection_source_fraction_0',\n",
      " '/quality_assessment/gt3l/signal_selection_source_fraction_1',\n",
      " '/quality_assessment/gt3l/signal_selection_source_fraction_2',\n",
      " '/quality_assessment/gt3l/signal_selection_source_fraction_3',\n",
      " '/quality_assessment/gt3r',\n",
      " '/quality_assessment/gt3r/delta_time',\n",
      " '/quality_assessment/gt3r/lat_mean',\n",
      " '/quality_assessment/gt3r/lon_mean',\n",
      " '/quality_assessment/gt3r/signal_selection_source_fraction_0',\n",
      " '/quality_assessment/gt3r/signal_selection_source_fraction_1',\n",
      " '/quality_assessment/gt3r/signal_selection_source_fraction_2',\n",
      " '/quality_assessment/gt3r/signal_selection_source_fraction_3']\n"
     ]
    }
   ],
   "source": [
    "pprint.pprint(variable_vals)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we can enter a list of variables to subset separated by comma using the `coverage` key. All forward slashes need to be included to indicate HDF group hierarchy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "coverage = '/ancillary_data/atlas_sdp_gps_epoch,\\\n",
    "/gt1l/land_ice_segments/atl06_quality_summary,\\\n",
    "/gt1l/land_ice_segments/delta_time,\\\n",
    "/gt1l/land_ice_segments/h_li,\\\n",
    "/gt1l/land_ice_segments/h_li_sigma,\\\n",
    "/gt1l/land_ice_segments/latitude,\\\n",
    "/gt1l/land_ice_segments/longitude,\\\n",
    "/gt1l/land_ice_segments/segment_id,\\\n",
    "/gt1l/land_ice_segments/sigma_geo_h,\\\n",
    "/gt1r/land_ice_segments/atl06_quality_summary,\\\n",
    "/gt1r/land_ice_segments/delta_time,\\\n",
    "/gt1r/land_ice_segments/h_li,\\\n",
    "/gt1r/land_ice_segments/h_li_sigma,\\\n",
    "/gt1r/land_ice_segments/latitude,\\\n",
    "/gt1r/land_ice_segments/longitude,\\\n",
    "/gt1r/land_ice_segments/segment_id,\\\n",
    "/gt1r/land_ice_segments/sigma_geo_h,\\\n",
    "/gt2l/land_ice_segments/atl06_quality_summary,\\\n",
    "/gt2l/land_ice_segments/delta_time,\\\n",
    "/gt2l/land_ice_segments/h_li,\\\n",
    "/gt2l/land_ice_segments/h_li_sigma,\\\n",
    "/gt2l/land_ice_segments/latitude,\\\n",
    "/gt2l/land_ice_segments/longitude,\\\n",
    "/gt2l/land_ice_segments/segment_id,\\\n",
    "/gt2l/land_ice_segments/sigma_geo_h,\\\n",
    "/gt2r/land_ice_segments/atl06_quality_summary,\\\n",
    "/gt2r/land_ice_segments/delta_time,\\\n",
    "/gt2r/land_ice_segments/h_li,\\\n",
    "/gt2r/land_ice_segments/h_li_sigma,\\\n",
    "/gt2r/land_ice_segments/latitude,\\\n",
    "/gt2r/land_ice_segments/longitude,\\\n",
    "/gt2r/land_ice_segments/segment_id,\\\n",
    "/gt2r/land_ice_segments/sigma_geo_h,\\\n",
    "/gt3l/land_ice_segments/atl06_quality_summary,\\\n",
    "/gt3l/land_ice_segments/delta_time,\\\n",
    "/gt3l/land_ice_segments/h_li,\\\n",
    "/gt3l/land_ice_segments/h_li_sigma,\\\n",
    "/gt3l/land_ice_segments/latitude,\\\n",
    "/gt3l/land_ice_segments/longitude,\\\n",
    "/gt3l/land_ice_segments/segment_id,\\\n",
    "/gt3l/land_ice_segments/sigma_geo_h,\\\n",
    "/gt3r/land_ice_segments/atl06_quality_summary,\\\n",
    "/gt3r/land_ice_segments/delta_time,\\\n",
    "/gt3r/land_ice_segments/h_li,\\\n",
    "/gt3r/land_ice_segments/h_li_sigma,\\\n",
    "/gt3r/land_ice_segments/latitude,\\\n",
    "/gt3r/land_ice_segments/longitude,\\\n",
    "/gt3r/land_ice_segments/segment_id,\\\n",
    "/gt3r/land_ice_segments/sigma_geo_h,\\\n",
    "/orbit_info/cycle_number,\\\n",
    "/orbit_info/rgt,\\\n",
    "/orbit_info/orbit_number' "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Request data from the NSIDC data access API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Set NSIDC data access base URL\n",
    "base_url = 'https://n5eil02u.ecs.nsidc.org/egi/request'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4\n"
     ]
    }
   ],
   "source": [
    "# Set number of granules requested per order, which we will initially set to 10.\n",
    "page_size = 10\n",
    "\n",
    "#Determine number of pages basd on page_size and total granules. Loop requests by this value\n",
    "page_num = math.ceil(len(granules)/page_size)\n",
    "\n",
    "#Set request mode. \n",
    "request_mode = 'async'\n",
    "\n",
    "# Determine how many individual orders we will request based on the number of granules requested\n",
    "\n",
    "print(page_num)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### After all of these KVP inputs, what does our request look like? Here's a summary of all possible KVPs that we explored, both for CMR searching and for the subsetter:\n",
    "\n",
    "#### CMR search keys:\n",
    "* `short_name=`\n",
    "* `version=`\n",
    "* `temporal=`\n",
    "* `bounding_box=`\n",
    "* `polygon=`\n",
    "\n",
    "#### Customization service keys:\n",
    "* `time=`\n",
    "* `bbox=`\n",
    "* `bounding_shape=` \n",
    "* `format=`\n",
    "* `projection=`\n",
    "* `projection_parameters=`\n",
    "* `Coverage=`\n",
    "\n",
    "#### No customization (access only):\n",
    "* `agent=`    \n",
    "* `include_meta=` \n",
    "    * `Y` by default. `N` for No metadata requested.\n",
    "\n",
    "#### Request configuration keys:\n",
    "* `request_mode=` \n",
    "* `page_size=`\n",
    "* `page_num=`\n",
    "* `token=`\n",
    "* `email=`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### If we were to create an API request based on our request parameters and submit into a web browser for example, here's what we end up with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'polygon' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-39-61722012488c>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m#Print API base URL + request parameters\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mAPI_request\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34mf'{base_url}?short_name={short_name}&version={latest_version}&temporal={temporal}&time={timevar}&polygon={polygon}&Coverage={coverage}&request_mode={request_mode}&page_size={page_size}&page_num={page_num}&token={token}&email={email}'\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      3\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mAPI_request\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'polygon' is not defined"
     ]
    }
   ],
   "source": [
    "#Print API base URL + request parameters\n",
    "API_request = f'{base_url}?short_name={short_name}&version={latest_version}&temporal={temporal}&time={timevar}&polygon={polygon}&Coverage={coverage}&request_mode={request_mode}&page_size={page_size}&page_num={page_num}&token={token}&email={email}'\n",
    "print(API_request)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### We'll also create a new dictionary of NSIDC API KVPs to be used in our subset request. Because we are looping through each page of requests, we'll add the `page_num` KVP to our dictionary within the loop below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'short_name': 'ATL06', 'version': '001', 'temporal': '2018-12-12T00:00:00Z,2019-02-28T23:59:59Z', 'bbox': '62.15,80.8,62.4,82.3', 'time': '2018-12-12T00:00:00,2019-02-28T23:59:59', 'Coverage': '/ancillary_data/atlas_sdp_gps_epoch,/gt1l/land_ice_segments/atl06_quality_summary,/gt1l/land_ice_segments/delta_time,/gt1l/land_ice_segments/h_li,/gt1l/land_ice_segments/h_li_sigma,/gt1l/land_ice_segments/latitude,/gt1l/land_ice_segments/longitude,/gt1l/land_ice_segments/segment_id,/gt1l/land_ice_segments/sigma_geo_h,/gt1r/land_ice_segments/atl06_quality_summary,/gt1r/land_ice_segments/delta_time,/gt1r/land_ice_segments/h_li,/gt1r/land_ice_segments/h_li_sigma,/gt1r/land_ice_segments/latitude,/gt1r/land_ice_segments/longitude,/gt1r/land_ice_segments/segment_id,/gt1r/land_ice_segments/sigma_geo_h,/gt2l/land_ice_segments/atl06_quality_summary,/gt2l/land_ice_segments/delta_time,/gt2l/land_ice_segments/h_li,/gt2l/land_ice_segments/h_li_sigma,/gt2l/land_ice_segments/latitude,/gt2l/land_ice_segments/longitude,/gt2l/land_ice_segments/segment_id,/gt2l/land_ice_segments/sigma_geo_h,/gt2r/land_ice_segments/atl06_quality_summary,/gt2r/land_ice_segments/delta_time,/gt2r/land_ice_segments/h_li,/gt2r/land_ice_segments/h_li_sigma,/gt2r/land_ice_segments/latitude,/gt2r/land_ice_segments/longitude,/gt2r/land_ice_segments/segment_id,/gt2r/land_ice_segments/sigma_geo_h,/gt3l/land_ice_segments/atl06_quality_summary,/gt3l/land_ice_segments/delta_time,/gt3l/land_ice_segments/h_li,/gt3l/land_ice_segments/h_li_sigma,/gt3l/land_ice_segments/latitude,/gt3l/land_ice_segments/longitude,/gt3l/land_ice_segments/segment_id,/gt3l/land_ice_segments/sigma_geo_h,/gt3r/land_ice_segments/atl06_quality_summary,/gt3r/land_ice_segments/delta_time,/gt3r/land_ice_segments/h_li,/gt3r/land_ice_segments/h_li_sigma,/gt3r/land_ice_segments/latitude,/gt3r/land_ice_segments/longitude,/gt3r/land_ice_segments/segment_id,/gt3r/land_ice_segments/sigma_geo_h,/orbit_info/cycle_number,/orbit_info/rgt,/orbit_info/orbit_number', 'request_mode': 'async', 'page_size': 10, 'token': '3CFC8D67-7A06-29C4-F5C1-5EB938C1E71C', 'email': 'jessica.scheick@maine.edu'}\n"
     ]
    }
   ],
   "source": [
    "subset_params = {\n",
    "    'short_name': short_name, \n",
    "    'version': latest_version, \n",
    "    'temporal': temporal, \n",
    "    'bbox': bbox,\n",
    "    'time': timevar,  \n",
    "    'Coverage': coverage, \n",
    "    'request_mode': request_mode, \n",
    "    'page_size': page_size,  \n",
    "    'token': token, \n",
    "    'email': email, \n",
    "    }\n",
    "print(subset_params)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### We'll request the same data but without any subsetting services applied. Let's create another request parameter dictionary with the `time` and `coverage` service keys removed, and we'll add `agent=NO` instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'short_name': 'ATL06', 'version': '001', 'temporal': '2018-12-12T00:00:00Z,2019-02-28T23:59:59Z', 'bounding_box': '62.15,80.8,62.4,82.3', 'agent': 'NO', 'include_meta': 'Y', 'request_mode': 'async', 'page_size': 10, 'token': '3CFC8D67-7A06-29C4-F5C1-5EB938C1E71C', 'email': 'jessica.scheick@maine.edu'}\n"
     ]
    }
   ],
   "source": [
    "request_params = {\n",
    "    'short_name': short_name, \n",
    "    'version': latest_version, \n",
    "    'temporal': temporal,\n",
    "    'bounding_box': bbox,\n",
    "    'agent' : 'NO',\n",
    "    'include_meta' : 'Y',\n",
    "    'request_mode': request_mode, \n",
    "    'page_size': page_size,  \n",
    "    'token': token, \n",
    "    'email': email, \n",
    "    }\n",
    "\n",
    "print(request_params)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Request Data\n",
    "\n",
    "#### Finally, we'll download the data directly to this notebook directory in a new Outputs folder. The progress of each order will be reported.\n",
    "\n",
    "We'll start by creating an output folder if the folder does not already exist."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = str(os.getcwd() + '/Outputs')\n",
    "if not os.path.exists(path):\n",
    "    os.mkdir(path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we'll submit our request without subsetting services:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Order:  1\n",
      "Request HTTP response:  201\n",
      "Order request URL:  https://n5eil02u.ecs.nsidc.org/egi/request?short_name=ATL06&version=001&temporal=2019-02-22T00%3A00%3A00Z%2C2019-02-22T23%3A59%3A59Z&polygon=-86.622742%2C-74.908126%2C-86.561712%2C-74.870913%2C-86.868859%2C-74.730522%2C-86.962905%2C-74.605038%2C-89.02594%2C-74.316754%2C-89.630517%2C-74.192147%2C-89.830808%2C-74.065919%2C-90.746478%2C-73.956258%2C-91.668214%2C-74.023169%2C-92.049815%2C-73.929387%2C-93.420791%2C-73.929327%2C-93.997163%2C-73.882768%2C-94.277701%2C-73.714183%2C-95.133017%2C-73.966355%2C-96.513501%2C-74.127404%2C-99.889802%2C-74.085347%2C-100.114438%2C-74.019422%2C-100.355131%2C-74.080906%2C-100.462734%2C-74.240864%2C-100.827076%2C-74.373988%2C-101.795349%2C-74.369597%2C-102.424826%2C-74.497263%2C-101.188725%2C-74.7179%2C-101.564382%2C-75.02971%2C-103.37484%2C-75.273725%2C-103.914847%2C-75.426057%2C-104.012128%2C-75.5223%2C-103.029452%2C-75.748774%2C-102.350567%2C-75.749245%2C-101.837882%2C-75.943066%2C-101.899461%2C-76.014086%2C-101.280944%2C-76.192769%2C-101.325735%2C-76.246168%2C-101.190803%2C-76.27106%2C-101.250474%2C-76.342292%2C-101.175067%2C-76.345822%2C-101.402436%2C-76.52035%2C-101.326063%2C-76.523929%2C-101.449791%2C-76.666392%2C-101.310795%2C-76.691373%2C-101.357407%2C-76.744819%2C-101.217404%2C-76.769752%2C-101.295133%2C-76.85887%2C-101.058051%2C-76.962123%2C-100.447336%2C-77.117686%2C-98.433698%2C-77.320866%2C-97.28308%2C-77.355688%2C-97.491148%2C-77.423178%2C-96.514174%2C-77.485919%2C-96.552494%2C-77.558236%2C-96.384656%2C-77.562336%2C-96.441516%2C-77.670857%2C-97.139363%2C-77.836566%2C-97.193451%2C-77.926901%2C-97.64271%2C-78.080044%2C-96.297869%2C-78.388943%2C-96.327803%2C-78.44329%2C-95.721466%2C-78.511065%2C-95.748962%2C-78.565482%2C-94.940425%2C-78.617072%2C-94.988611%2C-78.726066%2C-94.911669%2C-78.763976%2C-95.609268%2C-78.843079%2C-95.637038%2C-78.897535%2C-95.37191%2C-78.9391%2C-95.693408%2C-79.006456%2C-95.269903%2C-79.124145%2C-95.323729%2C-79.233172%2C-95.430206%2C-79.249633%2C-95.155505%2C-79.291032%2C-95.191045%2C-79.363748%2C-94.81352%2C-79.406486%2C-94.847075%2C-79.479253%2C-94.747448%2C-79.48078%2C-94.772403%2C-79.535367%2C-93.90411%2C-79.638844%2C-93.843651%2C-79.749409%2C-93.967323%2C-79.802836%2C-93.788723%2C-79.87821%2C-93.816393%2C-79.951128%2C-93.230546%2C-80.085534%2C-91.707475%2C-79.87748%2C-91.801545%2C-79.822143%2C-91.488897%2C-79.805457%2C-91.465152%2C-79.641131%2C-90.447349%2C-79.5894%2C-90.545492%2C-79.534464%2C-90.042319%2C-79.37062%2C-90.140775%2C-79.334083%2C-90.041814%2C-79.24285%2C-88.982186%2C-79.076903%2C-90.230262%2C-78.914333%2C-90.32191%2C-78.804808%2C-90.689626%2C-78.676516%2C-91.150024%2C-78.638589%2C-92.035347%2C-78.414844%2C-92.106013%2C-78.30491%2C-91.651645%2C-78.271472%2C-91.365784%2C-78.127206%2C-91.188783%2C-78.128018%2C-91.090167%2C-78.019109%2C-90.737076%2C-77.983849%2C-90.909191%2C-77.946905%2C-90.732603%2C-77.911009%2C-90.727088%2C-77.819973%2C-91.070502%2C-77.800626%2C-91.14118%2C-77.636469%2C-91.90279%2C-77.613923%2C-91.984627%2C-77.595116%2C-91.972963%2C-77.522365%2C-92.466819%2C-77.463587%2C-92.199521%2C-77.374914%2C-92.352136%2C-77.300761%2C-92.335283%2C-77.209895%2C-91.434206%2C-77.234653%2C-91.426015%2C-77.16193%2C-91.015545%2C-77.145686%2C-91.008355%2C-77.054784%2C-91.086397%2C-77.018096%2C-91.647835%2C-76.97871%2C-91.640906%2C-76.924199%2C-91.873848%2C-76.868024%2C-91.779021%2C-76.759619%2C-90.823937%2C-76.710073%2C-90.345113%2C-76.52953%2C-86.988029%2C-75.856983%2C-86.945563%2C-75.711143%2C-86.872234%2C-75.710165%2C-87.034102%2C-75.63967%2C-86.965004%2C-75.620616%2C-87.075115%2C-75.440545%2C-87.003154%2C-75.439609%2C-87.021872%2C-75.349129%2C-86.835058%2C-75.219586%2C-86.850654%2C-75.147247%2C-86.717729%2C-75.109052%2C-86.737771%2C-75.018662%2C-86.602149%2C-74.998483%2C-86.622742%2C-74.908126&agent=NO&include_meta=Y&request_mode=async&page_size=10&token=64085023-05AC-79C0-970D-E87478D7B3B3&email=jessica.scheick%40maine.edu&page_num=1\n",
      "Order request response XML content:  b'<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"yes\"?>\\n<eesi:agentResponse xsi:schemaLocation=\"http://eosdis.nasa.gov/esi/rsp/e https://newsroom.gsfc.nasa.gov/esi/8.1/schemas/ESIAgentResponseExternal.xsd\" xmlns=\"\" xmlns:iesi=\"http://eosdis.nasa.gov/esi/rsp/i\" xmlns:ssw=\"http://newsroom.gsfc.nasa.gov/esi/rsp/ssw\" xmlns:eesi=\"http://eosdis.nasa.gov/esi/rsp/e\" xmlns:esi=\"http://eosdis.nasa.gov/esi/rsp\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\">\\n    <order>\\n        <orderId>5000000319017</orderId>\\n        <Instructions>You may receive an email about your order if you specified an EMAIL address. &lt;br/&gt;&lt;br/&gt;The instructions used to process this order are:  Processing tool=NO. Include metadata and processing history=Y. Granule id(s)=SC:ATL06.001:161340025,SC:ATL06.001:161340094,SC:ATL06.001:161340262,SC:ATL06.001:161326920. Email address=jessica.scheick@maine.edu.</Instructions>\\n    </order>\\n    <contactInformation>\\n        <contactName>NSIDC User Services</contactName>\\n        <contactEmail>nsidc@nsidc.org</contactEmail>\\n    </contactInformation>\\n    <processInfo>\\n        <processDuration>PT0.120S</processDuration>\\n        <subagentId>NO</subagentId>\\n    </processInfo>\\n    <requestStatus>\\n        <status>pending</status>\\n        <numberProcessed>0</numberProcessed>\\n        <totalNumber>4</totalNumber>\\n    </requestStatus>\\n</eesi:agentResponse>\\n'\n",
      "order ID:  5000000319017\n",
      "status URL:  https://n5eil02u.ecs.nsidc.org/egi/request/5000000319017\n",
      "HTTP response from order response URL:  201\n",
      "Data request  1  is submitting...\n",
      "Initial request status is  processing\n",
      "Status is not complete. Trying again.\n",
      "Retry request status is:  complete\n",
      "Zip download URL:  https://n5eil02u.ecs.nsidc.org/esir/5000000319017.zip\n",
      "Beginning download of zipped output...\n",
      "Data request 1 is complete.\n"
     ]
    }
   ],
   "source": [
    "# Request data service for each page number, and unzip outputs\n",
    "\n",
    "for i in range(page_num):\n",
    "    page_val = i + 1\n",
    "    print('Order: ', page_val)\n",
    "    request_params.update( {'page_num': page_val} )\n",
    "    \n",
    "# For all requests other than spatial file upload, use get function\n",
    "    request = session.get(base_url, params=request_params)\n",
    "    \n",
    "    print('Request HTTP response: ', request.status_code)\n",
    "\n",
    "# Raise bad request: Loop will stop for bad response code.\n",
    "    request.raise_for_status()\n",
    "    print('Order request URL: ', request.url)\n",
    "    esir_root = ET.fromstring(request.content)\n",
    "    print('Order request response XML content: ', request.content)\n",
    "\n",
    "#Look up order ID\n",
    "    orderlist = []   \n",
    "    for order in esir_root.findall(\"./order/\"):\n",
    "        orderlist.append(order.text)\n",
    "    orderID = orderlist[0]\n",
    "    print('order ID: ', orderID)\n",
    "\n",
    "#Create status URL\n",
    "    statusURL = base_url + '/' + orderID\n",
    "    print('status URL: ', statusURL)\n",
    "\n",
    "#Find order status\n",
    "    request_response = session.get(statusURL)    \n",
    "    print('HTTP response from order response URL: ', request_response.status_code)\n",
    "    \n",
    "# Raise bad request: Loop will stop for bad response code.\n",
    "    request_response.raise_for_status()\n",
    "    request_root = ET.fromstring(request_response.content)\n",
    "    statuslist = []\n",
    "    for status in request_root.findall(\"./requestStatus/\"):\n",
    "        statuslist.append(status.text)\n",
    "    status = statuslist[0]\n",
    "    print('Data request ', page_val, ' is submitting...')\n",
    "    print('Initial request status is ', status)\n",
    "\n",
    "#Continue loop while request is still processing\n",
    "    while status == 'pending' or status == 'processing': \n",
    "        print('Status is not complete. Trying again.')\n",
    "        time.sleep(10)\n",
    "        loop_response = session.get(statusURL)\n",
    "\n",
    "# Raise bad request: Loop will stop for bad response code.\n",
    "        loop_response.raise_for_status()\n",
    "        loop_root = ET.fromstring(loop_response.content)\n",
    "\n",
    "#find status\n",
    "        statuslist = []\n",
    "        for status in loop_root.findall(\"./requestStatus/\"):\n",
    "            statuslist.append(status.text)\n",
    "        status = statuslist[0]\n",
    "        print('Retry request status is: ', status)\n",
    "        if status == 'pending' or status == 'processing':\n",
    "            continue\n",
    "\n",
    "#Order can either complete, complete_with_errors, or fail:\n",
    "# Provide complete_with_errors error message:\n",
    "    if status == 'complete_with_errors' or status == 'failed':\n",
    "        messagelist = []\n",
    "        for message in loop_root.findall(\"./processInfo/\"):\n",
    "            messagelist.append(message.text)\n",
    "        print('error messages:')\n",
    "        pprint.pprint(messagelist)\n",
    "\n",
    "# Download zipped order if status is complete or complete_with_errors\n",
    "    if status == 'complete' or status == 'complete_with_errors':\n",
    "        downloadURL = 'https://n5eil02u.ecs.nsidc.org/esir/' + orderID + '.zip'\n",
    "        print('Zip download URL: ', downloadURL)\n",
    "        print('Beginning download of zipped output...')\n",
    "        zip_response = session.get(downloadURL)\n",
    "        # Raise bad request: Loop will stop for bad response code.\n",
    "        zip_response.raise_for_status()\n",
    "        with zipfile.ZipFile(io.BytesIO(zip_response.content)) as z:\n",
    "            z.extractall(path)\n",
    "        print('Data request', page_val, 'is complete.')\n",
    "    else: print('Request failed.')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's run our request loop again, this time with subsetting services applied. We will post the KML file directly to the API:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'short_name': 'ATL06', 'version': '001', 'temporal': '2018-12-12T00:00:00Z,2019-02-28T23:59:59Z', 'bbox': '62.15,80.8,62.4,82.3', 'time': '2018-12-12T00:00:00,2019-02-28T23:59:59', 'Coverage': '/ancillary_data/atlas_sdp_gps_epoch,/gt1l/land_ice_segments/atl06_quality_summary,/gt1l/land_ice_segments/delta_time,/gt1l/land_ice_segments/h_li,/gt1l/land_ice_segments/h_li_sigma,/gt1l/land_ice_segments/latitude,/gt1l/land_ice_segments/longitude,/gt1l/land_ice_segments/segment_id,/gt1l/land_ice_segments/sigma_geo_h,/gt1r/land_ice_segments/atl06_quality_summary,/gt1r/land_ice_segments/delta_time,/gt1r/land_ice_segments/h_li,/gt1r/land_ice_segments/h_li_sigma,/gt1r/land_ice_segments/latitude,/gt1r/land_ice_segments/longitude,/gt1r/land_ice_segments/segment_id,/gt1r/land_ice_segments/sigma_geo_h,/gt2l/land_ice_segments/atl06_quality_summary,/gt2l/land_ice_segments/delta_time,/gt2l/land_ice_segments/h_li,/gt2l/land_ice_segments/h_li_sigma,/gt2l/land_ice_segments/latitude,/gt2l/land_ice_segments/longitude,/gt2l/land_ice_segments/segment_id,/gt2l/land_ice_segments/sigma_geo_h,/gt2r/land_ice_segments/atl06_quality_summary,/gt2r/land_ice_segments/delta_time,/gt2r/land_ice_segments/h_li,/gt2r/land_ice_segments/h_li_sigma,/gt2r/land_ice_segments/latitude,/gt2r/land_ice_segments/longitude,/gt2r/land_ice_segments/segment_id,/gt2r/land_ice_segments/sigma_geo_h,/gt3l/land_ice_segments/atl06_quality_summary,/gt3l/land_ice_segments/delta_time,/gt3l/land_ice_segments/h_li,/gt3l/land_ice_segments/h_li_sigma,/gt3l/land_ice_segments/latitude,/gt3l/land_ice_segments/longitude,/gt3l/land_ice_segments/segment_id,/gt3l/land_ice_segments/sigma_geo_h,/gt3r/land_ice_segments/atl06_quality_summary,/gt3r/land_ice_segments/delta_time,/gt3r/land_ice_segments/h_li,/gt3r/land_ice_segments/h_li_sigma,/gt3r/land_ice_segments/latitude,/gt3r/land_ice_segments/longitude,/gt3r/land_ice_segments/segment_id,/gt3r/land_ice_segments/sigma_geo_h,/orbit_info/cycle_number,/orbit_info/rgt,/orbit_info/orbit_number', 'request_mode': 'async', 'page_size': 10, 'token': '3CFC8D67-7A06-29C4-F5C1-5EB938C1E71C', 'email': 'jessica.scheick@maine.edu'}\n"
     ]
    }
   ],
   "source": [
    "print(subset_params)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Order:  1\n",
      "Request HTTP response:  201\n",
      "Order request URL:  https://n5eil02u.ecs.nsidc.org/egi/request?short_name=ATL06&version=001&temporal=2018-12-12T00%3A00%3A00Z%2C2019-02-28T23%3A59%3A59Z&bounding_box=62.15%2C80.8%2C62.4%2C82.3&agent=NO&include_meta=Y&request_mode=async&page_size=10&token=3CFC8D67-7A06-29C4-F5C1-5EB938C1E71C&email=jessica.scheick%40maine.edu\n",
      "Order request response XML content:  b'<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"yes\"?>\\n<eesi:agentResponse xsi:schemaLocation=\"http://eosdis.nasa.gov/esi/rsp/e https://newsroom.gsfc.nasa.gov/esi/8.1/schemas/ESIAgentResponseExternal.xsd\" xmlns=\"\" xmlns:iesi=\"http://eosdis.nasa.gov/esi/rsp/i\" xmlns:ssw=\"http://newsroom.gsfc.nasa.gov/esi/rsp/ssw\" xmlns:eesi=\"http://eosdis.nasa.gov/esi/rsp/e\" xmlns:esi=\"http://eosdis.nasa.gov/esi/rsp\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\">\\n    <order>\\n        <orderId>5000000319907</orderId>\\n        <Instructions>You may receive an email about your order if you specified an EMAIL address. &lt;br/&gt;&lt;br/&gt;The instructions used to process this order are:  Processing tool=NO. Include metadata and processing history=Y. Granule id(s)=SC:ATL06.001:161209848,SC:ATL06.001:161204276,SC:ATL06.001:161218204,SC:ATL06.001:161216171,SC:ATL06.001:161216792,SC:ATL06.001:161216948,SC:ATL06.001:161217541,SC:ATL06.001:161217775,SC:ATL06.001:161209824,SC:ATL06.001:161218943. Email address=jessica.scheick@maine.edu.</Instructions>\\n    </order>\\n    <contactInformation>\\n        <contactName>NSIDC User Services</contactName>\\n        <contactEmail>nsidc@nsidc.org</contactEmail>\\n    </contactInformation>\\n    <processInfo>\\n        <processDuration>PT0.097S</processDuration>\\n        <subagentId>NO</subagentId>\\n    </processInfo>\\n    <requestStatus>\\n        <status>processing</status>\\n        <numberProcessed>0</numberProcessed>\\n        <totalNumber>10</totalNumber>\\n    </requestStatus>\\n</eesi:agentResponse>\\n'\n",
      "order ID:  5000000319907\n",
      "status URL:  https://n5eil02u.ecs.nsidc.org/egi/request/5000000319907\n",
      "HTTP response from order response URL:  201\n",
      "Data request  1  is submitting...\n",
      "Initial request status is  processing\n",
      "Status is not complete. Trying again.\n",
      "Retry request status is:  complete\n",
      "Zip download URL:  https://n5eil02u.ecs.nsidc.org/esir/5000000319907.zip\n",
      "Beginning download of zipped output...\n",
      "Data request 1 is complete.\n",
      "Order:  2\n",
      "Request HTTP response:  201\n",
      "Order request URL:  https://n5eil02u.ecs.nsidc.org/egi/request?short_name=ATL06&version=001&temporal=2018-12-12T00%3A00%3A00Z%2C2019-02-28T23%3A59%3A59Z&bounding_box=62.15%2C80.8%2C62.4%2C82.3&agent=NO&include_meta=Y&request_mode=async&page_size=10&token=3CFC8D67-7A06-29C4-F5C1-5EB938C1E71C&email=jessica.scheick%40maine.edu\n",
      "Order request response XML content:  b'<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"yes\"?>\\n<eesi:agentResponse xsi:schemaLocation=\"http://eosdis.nasa.gov/esi/rsp/e https://newsroom.gsfc.nasa.gov/esi/8.1/schemas/ESIAgentResponseExternal.xsd\" xmlns=\"\" xmlns:iesi=\"http://eosdis.nasa.gov/esi/rsp/i\" xmlns:ssw=\"http://newsroom.gsfc.nasa.gov/esi/rsp/ssw\" xmlns:eesi=\"http://eosdis.nasa.gov/esi/rsp/e\" xmlns:esi=\"http://eosdis.nasa.gov/esi/rsp\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\">\\n    <order>\\n        <orderId>5000000319911</orderId>\\n        <Instructions>You may receive an email about your order if you specified an EMAIL address. &lt;br/&gt;&lt;br/&gt;The instructions used to process this order are:  Processing tool=NO. Include metadata and processing history=Y. Granule id(s)=SC:ATL06.001:161209848,SC:ATL06.001:161204276,SC:ATL06.001:161218204,SC:ATL06.001:161216171,SC:ATL06.001:161216792,SC:ATL06.001:161216948,SC:ATL06.001:161217541,SC:ATL06.001:161217775,SC:ATL06.001:161209824,SC:ATL06.001:161218943. Email address=jessica.scheick@maine.edu.</Instructions>\\n    </order>\\n    <contactInformation>\\n        <contactName>NSIDC User Services</contactName>\\n        <contactEmail>nsidc@nsidc.org</contactEmail>\\n    </contactInformation>\\n    <processInfo>\\n        <processDuration>PT0.098S</processDuration>\\n        <subagentId>NO</subagentId>\\n    </processInfo>\\n    <requestStatus>\\n        <status>processing</status>\\n        <numberProcessed>0</numberProcessed>\\n        <totalNumber>10</totalNumber>\\n    </requestStatus>\\n</eesi:agentResponse>\\n'\n",
      "order ID:  5000000319911\n",
      "status URL:  https://n5eil02u.ecs.nsidc.org/egi/request/5000000319911\n",
      "HTTP response from order response URL:  201\n",
      "Data request  2  is submitting...\n",
      "Initial request status is  processing\n",
      "Status is not complete. Trying again.\n",
      "Retry request status is:  complete\n",
      "Zip download URL:  https://n5eil02u.ecs.nsidc.org/esir/5000000319911.zip\n",
      "Beginning download of zipped output...\n",
      "Data request 2 is complete.\n",
      "Order:  3\n",
      "Request HTTP response:  201\n",
      "Order request URL:  https://n5eil02u.ecs.nsidc.org/egi/request?short_name=ATL06&version=001&temporal=2018-12-12T00%3A00%3A00Z%2C2019-02-28T23%3A59%3A59Z&bounding_box=62.15%2C80.8%2C62.4%2C82.3&agent=NO&include_meta=Y&request_mode=async&page_size=10&token=3CFC8D67-7A06-29C4-F5C1-5EB938C1E71C&email=jessica.scheick%40maine.edu\n",
      "Order request response XML content:  b'<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"yes\"?>\\n<eesi:agentResponse xsi:schemaLocation=\"http://eosdis.nasa.gov/esi/rsp/e https://newsroom.gsfc.nasa.gov/esi/8.1/schemas/ESIAgentResponseExternal.xsd\" xmlns=\"\" xmlns:iesi=\"http://eosdis.nasa.gov/esi/rsp/i\" xmlns:ssw=\"http://newsroom.gsfc.nasa.gov/esi/rsp/ssw\" xmlns:eesi=\"http://eosdis.nasa.gov/esi/rsp/e\" xmlns:esi=\"http://eosdis.nasa.gov/esi/rsp\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\">\\n    <order>\\n        <orderId>5000000319915</orderId>\\n        <Instructions>You may receive an email about your order if you specified an EMAIL address. &lt;br/&gt;&lt;br/&gt;The instructions used to process this order are:  Processing tool=NO. Include metadata and processing history=Y. Granule id(s)=SC:ATL06.001:161209848,SC:ATL06.001:161204276,SC:ATL06.001:161218204,SC:ATL06.001:161216171,SC:ATL06.001:161216792,SC:ATL06.001:161216948,SC:ATL06.001:161217541,SC:ATL06.001:161217775,SC:ATL06.001:161209824,SC:ATL06.001:161218943. Email address=jessica.scheick@maine.edu.</Instructions>\\n    </order>\\n    <contactInformation>\\n        <contactName>NSIDC User Services</contactName>\\n        <contactEmail>nsidc@nsidc.org</contactEmail>\\n    </contactInformation>\\n    <processInfo>\\n        <processDuration>PT0.106S</processDuration>\\n        <subagentId>NO</subagentId>\\n    </processInfo>\\n    <requestStatus>\\n        <status>processing</status>\\n        <numberProcessed>0</numberProcessed>\\n        <totalNumber>10</totalNumber>\\n    </requestStatus>\\n</eesi:agentResponse>\\n'\n",
      "order ID:  5000000319915\n",
      "status URL:  https://n5eil02u.ecs.nsidc.org/egi/request/5000000319915\n",
      "HTTP response from order response URL:  201\n",
      "Data request  3  is submitting...\n",
      "Initial request status is  processing\n",
      "Status is not complete. Trying again.\n",
      "Retry request status is:  complete\n",
      "Zip download URL:  https://n5eil02u.ecs.nsidc.org/esir/5000000319915.zip\n",
      "Beginning download of zipped output...\n",
      "Data request 3 is complete.\n",
      "Order:  4\n",
      "Request HTTP response:  201\n",
      "Order request URL:  https://n5eil02u.ecs.nsidc.org/egi/request?short_name=ATL06&version=001&temporal=2018-12-12T00%3A00%3A00Z%2C2019-02-28T23%3A59%3A59Z&bounding_box=62.15%2C80.8%2C62.4%2C82.3&agent=NO&include_meta=Y&request_mode=async&page_size=10&token=3CFC8D67-7A06-29C4-F5C1-5EB938C1E71C&email=jessica.scheick%40maine.edu\n",
      "Order request response XML content:  b'<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"yes\"?>\\n<eesi:agentResponse xsi:schemaLocation=\"http://eosdis.nasa.gov/esi/rsp/e https://newsroom.gsfc.nasa.gov/esi/8.1/schemas/ESIAgentResponseExternal.xsd\" xmlns=\"\" xmlns:iesi=\"http://eosdis.nasa.gov/esi/rsp/i\" xmlns:ssw=\"http://newsroom.gsfc.nasa.gov/esi/rsp/ssw\" xmlns:eesi=\"http://eosdis.nasa.gov/esi/rsp/e\" xmlns:esi=\"http://eosdis.nasa.gov/esi/rsp\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\">\\n    <order>\\n        <orderId>5000000319918</orderId>\\n        <Instructions>You may receive an email about your order if you specified an EMAIL address. &lt;br/&gt;&lt;br/&gt;The instructions used to process this order are:  Processing tool=NO. Include metadata and processing history=Y. Granule id(s)=SC:ATL06.001:161209848,SC:ATL06.001:161204276,SC:ATL06.001:161218204,SC:ATL06.001:161216171,SC:ATL06.001:161216792,SC:ATL06.001:161216948,SC:ATL06.001:161217541,SC:ATL06.001:161217775,SC:ATL06.001:161209824,SC:ATL06.001:161218943. Email address=jessica.scheick@maine.edu.</Instructions>\\n    </order>\\n    <contactInformation>\\n        <contactName>NSIDC User Services</contactName>\\n        <contactEmail>nsidc@nsidc.org</contactEmail>\\n    </contactInformation>\\n    <processInfo>\\n        <processDuration>PT0.100S</processDuration>\\n        <subagentId>NO</subagentId>\\n    </processInfo>\\n    <requestStatus>\\n        <status>processing</status>\\n        <numberProcessed>0</numberProcessed>\\n        <totalNumber>10</totalNumber>\\n    </requestStatus>\\n</eesi:agentResponse>\\n'\n",
      "order ID:  5000000319918\n",
      "status URL:  https://n5eil02u.ecs.nsidc.org/egi/request/5000000319918\n",
      "HTTP response from order response URL:  201\n",
      "Data request  4  is submitting...\n",
      "Initial request status is  processing\n",
      "Status is not complete. Trying again.\n",
      "Retry request status is:  complete\n",
      "Zip download URL:  https://n5eil02u.ecs.nsidc.org/esir/5000000319918.zip\n",
      "Beginning download of zipped output...\n",
      "Data request 4 is complete.\n"
     ]
    }
   ],
   "source": [
    "# Request data service for each page number, and unzip outputs\n",
    "\n",
    "for i in range(page_num):\n",
    "    page_val = i + 1\n",
    "    print('Order: ', page_val)\n",
    "    subset_params.update( {'page_num': page_val} )\n",
    "    \n",
    "# Post polygon to API endpoint for polygon subsetting to subset based on original, non-simplified KML file\n",
    "\n",
    "#    shape_post = {'shapefile': open(kml_filepath, 'rb')}\n",
    "#    request = session.post(base_url, params=subset_params, files=shape_post) \n",
    "    \n",
    "# FOR ALL OTHER REQUESTS THAT DO NOT UTILIZED AN UPLOADED POLYGON FILE, USE A GET REQUEST INSTEAD OF POST:\n",
    "    request = session.get(base_url, params=request_params)\n",
    "    \n",
    "    print('Request HTTP response: ', request.status_code)\n",
    "\n",
    "# Raise bad request: Loop will stop for bad response code.\n",
    "    request.raise_for_status()\n",
    "    print('Order request URL: ', request.url)\n",
    "    esir_root = ET.fromstring(request.content)\n",
    "    print('Order request response XML content: ', request.content)\n",
    "\n",
    "# Look up order ID\n",
    "    orderlist = []   \n",
    "    for order in esir_root.findall(\"./order/\"):\n",
    "        orderlist.append(order.text)\n",
    "    orderID = orderlist[0]\n",
    "    print('order ID: ', orderID)\n",
    "\n",
    "# Create status URL\n",
    "    statusURL = base_url + '/' + orderID\n",
    "    print('status URL: ', statusURL)\n",
    "\n",
    "# Find order status\n",
    "    request_response = session.get(statusURL)    \n",
    "    print('HTTP response from order response URL: ', request_response.status_code)\n",
    "    \n",
    "# Raise bad request: Loop will stop for bad response code.\n",
    "    request_response.raise_for_status()\n",
    "    request_root = ET.fromstring(request_response.content)\n",
    "    statuslist = []\n",
    "    for status in request_root.findall(\"./requestStatus/\"):\n",
    "        statuslist.append(status.text)\n",
    "    status = statuslist[0]\n",
    "    print('Data request ', page_val, ' is submitting...')\n",
    "    print('Initial request status is ', status)\n",
    "\n",
    "# Continue to loop while request is still processing\n",
    "    while status == 'pending' or status == 'processing': \n",
    "        print('Status is not complete. Trying again.')\n",
    "        time.sleep(10)\n",
    "        loop_response = session.get(statusURL)\n",
    "\n",
    "# Raise bad request: Loop will stop for bad response code.\n",
    "        loop_response.raise_for_status()\n",
    "        loop_root = ET.fromstring(loop_response.content)\n",
    "\n",
    "# Find status\n",
    "        statuslist = []\n",
    "        for status in loop_root.findall(\"./requestStatus/\"):\n",
    "            statuslist.append(status.text)\n",
    "        status = statuslist[0]\n",
    "        print('Retry request status is: ', status)\n",
    "        if status == 'pending' or status == 'processing':\n",
    "            continue\n",
    "\n",
    "# Order can either complete, complete_with_errors, or fail:\n",
    "# Provide complete_with_errors error message:\n",
    "    if status == 'complete_with_errors' or status == 'failed':\n",
    "        messagelist = []\n",
    "        for message in loop_root.findall(\"./processInfo/\"):\n",
    "            messagelist.append(message.text)\n",
    "        print('error messages:')\n",
    "        pprint.pprint(messagelist)\n",
    "\n",
    "# Download zipped order if status is complete or complete_with_errors\n",
    "    if status == 'complete' or status == 'complete_with_errors':\n",
    "        downloadURL = 'https://n5eil02u.ecs.nsidc.org/esir/' + orderID + '.zip'\n",
    "        print('Zip download URL: ', downloadURL)\n",
    "        print('Beginning download of zipped output...')\n",
    "        zip_response = session.get(downloadURL)\n",
    "        # Raise bad request: Loop will stop for bad response code.\n",
    "        zip_response.raise_for_status()\n",
    "        with zipfile.ZipFile(io.BytesIO(zip_response.content)) as z:\n",
    "            z.extractall(path)\n",
    "        print('Data request', page_val, 'is complete.')\n",
    "    else: print('Request failed.')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Clean up the Output folder by removing individual order folders:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Clean up Outputs folder by removing individual granule folders \n",
    "\n",
    "for root, dirs, files in os.walk(path, topdown=False):\n",
    "    for file in files:\n",
    "        try:\n",
    "            shutil.move(os.path.join(root, file), path)\n",
    "        except OSError:\n",
    "            pass\n",
    "        \n",
    "for root, dirs, files in os.walk(path):\n",
    "    for name in dirs:\n",
    "        os.rmdir(os.path.join(root, name))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['ATL06_20181212000549_11360102_001_01.h5',\n",
       " 'ATL06_20181212000549_11360102_001_01.iso.xml',\n",
       " 'ATL06_20181212001420_11360103_001_01.h5',\n",
       " 'ATL06_20181212001420_11360103_001_01.iso.xml',\n",
       " 'ATL06_20181212001945_11360104_001_01.h5',\n",
       " 'ATL06_20181212001945_11360104_001_01.iso.xml',\n",
       " 'ATL06_20181212002454_11360105_001_01.h5',\n",
       " 'ATL06_20181212002454_11360105_001_01.iso.xml',\n",
       " 'ATL06_20181212003019_11360106_001_01.h5',\n",
       " 'ATL06_20181212003019_11360106_001_01.iso.xml',\n",
       " 'ATL06_20181212005855_11360110_001_01.h5',\n",
       " 'ATL06_20181212005855_11360110_001_01.iso.xml',\n",
       " 'ATL06_20181212010636_11360111_001_01.h5',\n",
       " 'ATL06_20181212010636_11360111_001_01.iso.xml',\n",
       " 'ATL06_20181212012000_11360113_001_01.h5',\n",
       " 'ATL06_20181212012000_11360113_001_01.iso.xml',\n",
       " 'ATL06_20181212014007_11370102_001_01.h5',\n",
       " 'ATL06_20181212014007_11370102_001_01.iso.xml',\n",
       " 'ATL06_20181212014837_11370103_001_01.h5',\n",
       " 'ATL06_20181212014837_11370103_001_01.iso.xml']"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#List files\n",
    "sorted(os.listdir(path))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you're interested in the streaming request method, an example loop is below: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set page size to 1 to improve performance\n",
    "page_size = 1\n",
    "request_params.update( {'page_size': page_size})\n",
    "\n",
    "# No metadata to only return a single output\n",
    "request_params.update( {'include_meta': 'N'})\n",
    "\n",
    "#Determine number of pages basd on page_size and total granules. Loop requests by this value\n",
    "page_num = math.ceil(len(granules)/page_size)\n",
    "print(page_num)\n",
    "\n",
    "#Set request mode. \n",
    "request_params.update( {'request_mode': 'stream'})\n",
    "\n",
    "print(request_params)\n",
    "\n",
    "os.chdir(path)\n",
    "\n",
    "for i in range(page_num):\n",
    "    page_val = i + 1\n",
    "    print('Order: ', page_val)\n",
    "    request_params.update( {'page_num': page_val})\n",
    "    request = session.get(base_url, params=request_params)\n",
    "    print('HTTP response from order response URL: ', request.status_code)\n",
    "    request.raise_for_status()\n",
    "    d = request.headers['content-disposition']\n",
    "    fname = re.findall('filename=(.+)', d)\n",
    "    open(eval(fname[0]), 'wb').write(request.content)\n",
    "    print('Data request', page_val, 'is complete.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Visualization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import geopandas as gpd\n",
    "import h5py\n",
    "import glob\n",
    "import os\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Magic function to enable interactive plotting (zoom/pan) in Jupyter notebook\n",
    "#If running locally, this would be `%matplotlib notebook`, but since we're using Juptyerlab, we use widget\n",
    "%matplotlib widget\n",
    "#%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "import h5py\n",
    "import numpy as np\n",
    "\n",
    "def read_atl06(fname, bbox=None):\n",
    "    \"\"\" \n",
    "    Read 1 ATL06 file and output 6 reduced files. \n",
    "    \n",
    "    Extract variables of interest and separate the ATL06 file \n",
    "    into each beam (ground track) and ascending/descending orbits.\n",
    "    \"\"\"\n",
    "\n",
    "    # Each beam is a group\n",
    "    group = ['/gt1l', '/gt1r', '/gt2l', '/gt2r', '/gt3l', '/gt3r']\n",
    "\n",
    "    # Loop trough beams\n",
    "    for k,g in enumerate(group):\n",
    "    \n",
    "        #-----------------------------------#\n",
    "        # 1) Read in data for a single beam #\n",
    "        #-----------------------------------#\n",
    "    \n",
    "        # Load variables into memory (more can be added!)\n",
    "        with h5py.File(fname, 'r') as fi:\n",
    "            x_atc = fi[g+'/land_ice_segments/ground_track/x_atc'][:]\n",
    "            atl06_quality_summary = fi[g+'/land_ice_segments/atl06_quality_summary'][:]\n",
    "            \n",
    "            lat = fi[g+'/land_ice_segments/latitude'][:]\n",
    "            lon = fi[g+'/land_ice_segments/longitude'][:]\n",
    "            h_li = fi[g+'/land_ice_segments/h_li'][:]\n",
    "            s_li = fi[g+'/land_ice_segments/h_li_sigma'][:]\n",
    "            t_dt = fi[g+'/land_ice_segments/delta_time'][:]\n",
    "            q_flag = fi[g+'/land_ice_segments/atl06_quality_summary'][:]\n",
    "            s_fg = fi[g+'/land_ice_segments/fit_statistics/signal_selection_source'][:]\n",
    "            snr = fi[g+'/land_ice_segments/fit_statistics/snr_significance'][:]\n",
    "            h_rb = fi[g+'/land_ice_segments/fit_statistics/h_robust_sprd'][:]\n",
    "            dac = fi[g+'/land_ice_segments/geophysical/dac'][:]\n",
    "            f_sn = fi[g+'/land_ice_segments/geophysical/bsnow_conf'][:]\n",
    "            dh_fit_dx = fi[g+'/land_ice_segments/fit_statistics/dh_fit_dx'][:]\n",
    "            tide_earth = fi[g+'/land_ice_segments/geophysical/tide_earth'][:]\n",
    "            tide_load = fi[g+'/land_ice_segments/geophysical/tide_load'][:]\n",
    "            tide_ocean = fi[g+'/land_ice_segments/geophysical/tide_ocean'][:]\n",
    "            tide_pole = fi[g+'/land_ice_segments/geophysical/tide_pole'][:]\n",
    "            t_ref = fi['/ancillary_data/atlas_sdp_gps_epoch'][:]\n",
    "            rgt = fi['/orbit_info/rgt'][:] * np.ones(len(lat))\n",
    "            orb = np.full_like(h_li, k)\n",
    "\n",
    "        #---------------------------------------------#\n",
    "        # 2) Filter data according region and quality #\n",
    "        #---------------------------------------------#\n",
    "        \n",
    "        # Select a region of interest\n",
    "        if bbox:\n",
    "            lonmin, lonmax, latmin, latmax = bbox\n",
    "            bbox_mask = (lon >= lonmin) & (lon <= lonmax) & \\\n",
    "                        (lat >= latmin) & (lat <= latmax)\n",
    "        else:\n",
    "            bbox_mask = np.ones_like(lat, dtype=bool)  # get all\n",
    "            \n",
    "        # Only keep good data, and data inside bbox\n",
    "        mask = (q_flag == 0) & (np.abs(h_li) < 10e3) & (bbox_mask == 1)\n",
    "        \n",
    "        # Update variables\n",
    "        lat, lon, h_li, s_li, t_dt, h_rb, s_fg, snr, q_flag, f_sn, \\\n",
    "            tide_earth, tide_load, tide_ocean, tide_pole, dac, rgt, orb, x_atc, atl06_quality_summary = \\\n",
    "                lat[mask], lon[mask], h_li[mask], s_li[mask], t_dt[mask], \\\n",
    "                h_rb[mask], s_fg[mask], snr[mask], q_flag[mask], f_sn[mask], \\\n",
    "                tide_earth[mask], tide_load[mask], tide_ocean[mask], \\\n",
    "                tide_pole[mask], dac[mask], rgt[mask], orb[mask], x_atc[mask], atl06_quality_summary[mask]\n",
    "\n",
    "        # Test for no data\n",
    "        if len(h_li) == 0: continue\n",
    "\n",
    "        #-------------------------------------#\n",
    "        # 3) Convert time and separate tracks #\n",
    "        #-------------------------------------#\n",
    "        \n",
    "        # Time in GPS seconds (secs sinde 1980...)\n",
    "        t_gps = t_ref + t_dt\n",
    "\n",
    "        # Time in decimal years\n",
    "        t_year = gps2dyr(t_gps)\n",
    "\n",
    "        # Determine orbit type\n",
    "        i_asc, i_des = track_type(t_year, lat)\n",
    "        \n",
    "        #-----------------------#\n",
    "        # 4) Save selected data #\n",
    "        #-----------------------#\n",
    "        \n",
    "        # Define output file name\n",
    "        ofile = fname.replace('.h5', '_'+g[1:]+'.h5')\n",
    "                \n",
    "        # Save variables\n",
    "        with h5py.File(ofile, 'w') as f:\n",
    "            f['orbit'] = orb\n",
    "            f['lon'] = lon\n",
    "            f['lat'] = lat\n",
    "            f['h_elv'] = h_li\n",
    "            f['t_year'] = t_year\n",
    "            f['t_sec'] = t_gps\n",
    "            f['s_elv'] = s_li\n",
    "            f['h_rb'] = h_rb\n",
    "            f['s_fg'] = s_fg\n",
    "            f['snr'] = snr\n",
    "            f['q_flg'] = q_flag\n",
    "            f['f_sn'] = f_sn\n",
    "            f['tide_load'] = tide_load\n",
    "            f['tide_ocean'] = tide_ocean\n",
    "            f['tide_pole'] = tide_pole\n",
    "            f['tide_earth'] = tide_earth\n",
    "            f['dac'] = dac\n",
    "            f['rgt'] = rgt\n",
    "            f['trk_type'] = i_asc\n",
    "            f['x_atc'] = x_atc\n",
    "            f['atl06_quality_summary'] = atl06_quality_summary\n",
    "\n",
    "            print('out ->', ofile)\n",
    "                "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def list_files_local(path):\n",
    "    \"\"\" Get file list form local folder. \"\"\"\n",
    "    from glob import glob\n",
    "    return glob(path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pyproj\n",
    "from astropy.time import Time\n",
    "\n",
    "def gps2dyr(time):\n",
    "    \"\"\" Converte GPS time to decimal years. \"\"\"\n",
    "    return Time(time, format='gps').decimalyear\n",
    "\n",
    "\n",
    "def track_type(time, lat, tmax=1):\n",
    "    \"\"\"\n",
    "    Separate tracks into ascending and descending.\n",
    "    \n",
    "    Defines tracks as segments with time breaks > tmax,\n",
    "    and tests whether lat increases or decreases w/time.\n",
    "    \"\"\"\n",
    "    tracks = np.zeros(lat.shape)  # generate track segment\n",
    "    tracks[0:np.argmax(np.abs(lat))] = 1  # set values for segment\n",
    "    i_asc = np.zeros(tracks.shape, dtype=bool)  # output index array\n",
    "\n",
    "    # Loop trough individual secments\n",
    "    for track in np.unique(tracks):\n",
    "    \n",
    "        i_track, = np.where(track == tracks)  # get all pts from seg\n",
    "    \n",
    "        if len(i_track) < 2: continue\n",
    "    \n",
    "        # Test if lat increases (asc) or decreases (des) w/time\n",
    "        i_min = time[i_track].argmin()\n",
    "        i_max = time[i_track].argmax()\n",
    "        lat_diff = lat[i_track][i_max] - lat[i_track][i_min]\n",
    "    \n",
    "        # Determine track type\n",
    "        if lat_diff > 0:  i_asc[i_track] = True\n",
    "    \n",
    "    return i_asc, np.invert(i_asc)  # index vectors\n",
    "\n",
    "\n",
    "def transform_coord(proj1, proj2, x, y):\n",
    "    \"\"\"\n",
    "    Transform coordinates from proj1 to proj2 (EPSG num).\n",
    "\n",
    "    Example EPSG projs:\n",
    "        Geodetic (lon/lat): 4326\n",
    "        Polar Stereo AnIS (x/y): 3031\n",
    "        Polar Stereo GrIS (x/y): 3413\n",
    "    \"\"\"\n",
    "    # Set full EPSG projection strings\n",
    "    proj1 = pyproj.Proj(\"+init=EPSG:\"+str(proj1))\n",
    "    proj2 = pyproj.Proj(\"+init=EPSG:\"+str(proj2))\n",
    "    return pyproj.transform(proj1, proj2, x, y)  # convert\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def read_h5(fname, vnames=[]):\n",
    "    \"\"\" Simple HDF5 reader. \"\"\"\n",
    "    with h5py.File(fname, 'r') as f:\n",
    "        return [f[v][:] for v in vnames]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20181226065456_13540104_001_01_gt1l.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20181226065456_13540104_001_01_gt1r.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20181226065456_13540104_001_01_gt2l.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20181226065456_13540104_001_01_gt2r.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20181226065456_13540104_001_01_gt3l.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20181226065456_13540104_001_01_gt3r.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20181214212829_11800104_001_01_gt1l.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20181214212829_11800104_001_01_gt1r.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20181214212829_11800104_001_01_gt2l.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20181214212829_11800104_001_01_gt2r.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20181214212829_11800104_001_01_gt3l.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20181214212829_11800104_001_01_gt3r.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20190103063820_00890204_001_01_gt1l.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20190103063820_00890204_001_01_gt1r.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20190103063820_00890204_001_01_gt2l.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20190103063820_00890204_001_01_gt2r.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20190103063820_00890204_001_01_gt3l.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20190103063820_00890204_001_01_gt3r.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20181231202937_00520204_001_01_gt1l.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20181231202937_00520204_001_01_gt1r.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20181231202937_00520204_001_01_gt2l.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20181231202937_00520204_001_01_gt2r.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20181231202937_00520204_001_01_gt3l.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20181231202937_00520204_001_01_gt3r.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20181213074532_11560104_001_01_gt1l.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20181213074532_11560104_001_01_gt1r.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20181213074532_11560104_001_01_gt2l.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20181213074532_11560104_001_01_gt2r.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20181213074532_11560104_001_01_gt3l.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20181213074532_11560104_001_01_gt3r.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20190104202117_01130204_001_01_gt1l.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20190104202117_01130204_001_01_gt1r.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20190104202117_01130204_001_01_gt2l.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20190104202117_01130204_001_01_gt2r.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20190104202117_01130204_001_01_gt3l.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20190104202117_01130204_001_01_gt3r.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20181223204614_13170104_001_01_gt1l.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20181223204614_13170104_001_01_gt1r.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20181223204614_13170104_001_01_gt2l.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20181223204614_13170104_001_01_gt2r.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20181223204614_13170104_001_01_gt3l.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20181223204614_13170104_001_01_gt3r.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20181230064639_00280204_001_01_gt1l.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20181230064639_00280204_001_01_gt1r.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20181230064639_00280204_001_01_gt2l.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20181230064639_00280204_001_01_gt2r.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20181230064639_00280204_001_01_gt3l.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20181230064639_00280204_001_01_gt3r.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20181227203753_13780104_001_01_gt1r.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20181227203753_13780104_001_01_gt2l.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20181227203753_13780104_001_01_gt2r.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20181227203753_13780104_001_01_gt3l.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20181227203753_13780104_001_01_gt3r.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20181222070316_12930104_001_01_gt1l.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20181222070316_12930104_001_01_gt1r.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20181222070316_12930104_001_01_gt2l.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20181222070316_12930104_001_01_gt2r.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20181222070316_12930104_001_01_gt3l.h5\n",
      "out -> /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20181222070316_12930104_001_01_gt3r.h5\n"
     ]
    }
   ],
   "source": [
    "files = list_files_local('/home/jovyan/WaterNoice/ice-ocean_data/*_01.h5')\n",
    "\n",
    "for fname in files:\n",
    "    read_atl06(fname)\n",
    "    #!h5ls -r fname"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/                        Group\n",
      "/atl06_quality_summary   Dataset {588}\n",
      "/dac                     Dataset {588}\n",
      "/f_sn                    Dataset {588}\n",
      "/h_elv                   Dataset {588}\n",
      "/h_rb                    Dataset {588}\n",
      "/lat                     Dataset {588}\n",
      "/lon                     Dataset {588}\n",
      "/orbit                   Dataset {588}\n",
      "/q_flg                   Dataset {588}\n",
      "/rgt                     Dataset {588}\n",
      "/s_elv                   Dataset {588}\n",
      "/s_fg                    Dataset {588}\n",
      "/snr                     Dataset {588}\n",
      "/t_sec                   Dataset {588}\n",
      "/t_year                  Dataset {588}\n",
      "/tide_earth              Dataset {588}\n",
      "/tide_load               Dataset {588}\n",
      "/tide_ocean              Dataset {588}\n",
      "/tide_pole               Dataset {588}\n",
      "/trk_type                Dataset {588}\n",
      "/x_atc                   Dataset {588}\n"
     ]
    }
   ],
   "source": [
    "!h5ls -r /home/jovyan/WaterNoice/ice-ocean_data/ATL06_20181222070316_12930104_001_01_gt3r.h5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ec628956f22048eaaec6bf7e34344f51",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FigureCanvasNbAgg()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "files = list_files_local('/home/jovyan/WaterNoice/ice-ocean_data/*gt1l.h5')\n",
    "\n",
    "fig = plt.figure(figsize=(14,5))\n",
    "ax1 = plt.subplot(121)\n",
    "ax2 = plt.subplot(122)\n",
    "\n",
    "for fname in files:\n",
    "\n",
    "    \n",
    "    lat, lon, x_atc, t, h = read_h5(fname, ['lat','lon', 'x_atc', 't_year', 'h_elv'])\n",
    "    #x, y = transform_coord(4326, 3031, lon, lat)\n",
    "    #dist = np.sqrt((x-x[0])**2 + (y-y[0])**2)\n",
    " \n",
    "    ax1.plot(x_atc, h, '.', label='%s' % fname[-7:-3])\n",
    "    ax2.plot(lat,lon, '.', label='%s' % fname[-7:-3])\n",
    "    \n",
    "ax1.set_xlabel('along track distance (m)')\n",
    "ax1.set_ylabel('height (m)')\n",
    "ax1.legend()\n",
    "ax2.set_xlabel('time (yr)')\n",
    "ax2.set_ylabel('height (m)')\n",
    "ax2.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pyproj\n",
    "from astropy.time import Time\n",
    "\n",
    "def gps2dyr(time):\n",
    "    \"\"\" Converte GPS time to decimal years. \"\"\"\n",
    "    return Time(time, format='gps').decimalyear\n",
    "\n",
    "\n",
    "def track_type(time, lat, tmax=1):\n",
    "    \"\"\"\n",
    "    Separate tracks into ascending and descending.\n",
    "    \n",
    "    Defines tracks as segments with time breaks > tmax,\n",
    "    and tests whether lat increases or decreases w/time.\n",
    "    \"\"\"\n",
    "    tracks = np.zeros(lat.shape)  # generate track segment\n",
    "    tracks[0:np.argmax(np.abs(lat))] = 1  # set values for segment\n",
    "    i_asc = np.zeros(tracks.shape, dtype=bool)  # output index array\n",
    "\n",
    "    # Loop trough individual secments\n",
    "    for track in np.unique(tracks):\n",
    "    \n",
    "        i_track, = np.where(track == tracks)  # get all pts from seg\n",
    "    \n",
    "        if len(i_track) < 2: continue\n",
    "    \n",
    "        # Test if lat increases (asc) or decreases (des) w/time\n",
    "        i_min = time[i_track].argmin()\n",
    "        i_max = time[i_track].argmax()\n",
    "        lat_diff = lat[i_track][i_max] - lat[i_track][i_min]\n",
    "    \n",
    "        # Determine track type\n",
    "        if lat_diff > 0:  i_asc[i_track] = True\n",
    "    \n",
    "    return i_asc, np.invert(i_asc)  # index vectors\n",
    "\n",
    "\n",
    "def transform_coord(proj1, proj2, x, y):\n",
    "    \"\"\"\n",
    "    Transform coordinates from proj1 to proj2 (EPSG num).\n",
    "\n",
    "    Example EPSG projs:\n",
    "        Geodetic (lon/lat): 4326\n",
    "        Polar Stereo AnIS (x/y): 3031\n",
    "        Polar Stereo GrIS (x/y): 3413\n",
    "    \"\"\"\n",
    "    # Set full EPSG projection strings\n",
    "    proj1 = pyproj.Proj(\"+init=EPSG:\"+str(proj1))\n",
    "    proj2 = pyproj.Proj(\"+init=EPSG:\"+str(proj2))\n",
    "    return pyproj.transform(proj1, proj2, x, y)  # convert\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
